Hetzner Private Cloud
In this technical retrospective (2017), @stemuk shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Cloud & Mobile Era (2014–2019)
COMMUNITY SCORE 60 points
TOTAL DISCUSSIONS 83 comments
I used Hetzner services until I received an email from there about some of their servers being infected in the RAM.
Here is the email that I received in 2013:
Dear Client
At the end of last week, Hetzner technicians discovered a "backdoor" in one of our internal monitoring systems (Nagios).
An investigation was launched immediately and showed that the administration interface for dedicated root servers (Robot) had also been affected. Current findings would suggest that fragments of our client database had been copied externally.
As a result, we currently have to consider the client data stored in our Robot as compromised.
To our knowledge, the malicious program that we have discovered is as yet unknown and has never appeared before.
The malicious code used in the "backdoor" exclusively infects the RAM. First analysis suggests that the malicious code directly infiltrates running Apache and sshd processes. Here, the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected.
The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code.
We have commissioned an external security company with a detailed analysis of the incident to support our in-house administrators. At this stage, analysis of the incident has not yet been completed.
The access passwords for your Robot client account are stored in our database as Hash (SHA256) with salt. As a precaution, we recommend that you change your client passwords in the Robot.
With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.
Hetzner technicians are permanently working on localising and preventing possible security vulnerabilities as well as ensuring that our systems and infrastructure are kept as safe as possible. Data security is a very high priority for us. To expedite clarification further, we have reported this incident to the data security authority concerned.
Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in regard to this incident.
Naturally, we shall inform you of new developments immediately.
We very much regret this incident and thank you for your understanding and trust in us.
A special FAQs page has been set up at http://wiki.hetzner.de/index.php/Security_Issue/en ↗ to assist you with further enquiries.
Kind regards
Martin Hetzner
I used Hetzner like 10 years ago and didn't had good experience. I doubt anything changed over time. They offer home grade hardware and low priority support in return of cheap prices. They only deal with hardware and network issues (which is quite normal for Dedi service with normal support option)
- Hardware they offer is more prone to fail because of using home grade hardware for long time (especially HDD).
- It's almost impossible to convince them HDD is failing even with showing SMART logs. Hardware needs to fail so they will replace it.
- Hardware replacement times are quite fast (thanks to SLA). They replace it with another used HDD, if you want something newer, than they ask some money for replacing with less used HDD.
- They scan their network regularly for hosted malware, trojan etc. so if one of your sites get hijacked and has iframe viruses etc. Hetzner will null route your server.
- If your IP gets DDOS, null route.
- If you get DMCA warning, null route without waiting 24 hours.
- If your NAT leaks your internal traffic to WLAN, null route.
- It takes almost few day to lift null route ban on your server when you get in contact with support. It's okay for support tickets to wait in queue for long time because of service level but I believe null route tickets needs priority no matter what.
We decided to move over to another provider after having problems.
Hetzner also owns few other brands like Serverloft.
We have a lot of experience with Hetzner servers and I used and recommend this hoster since at least 10 years intensively with tens and hundreds of servers.
The most important thing with Hetzner Servers is to monitor everything very closely:
- CPU Temperature
- RAM
- Disks (SMART)
- Software- and Hardware raids
- Network (interface) errors
The servers are usually consumer-grade hardware components which have more often issues under heavy load so you have to expect down-times and broken components. However, if you are aware of that and you can easily shift around that with your software Hetzner will save you serious money (10 to 15 times cheaper than GCP and AWS). Also to mention is that their customer support is first class if you tell them all required details and exactly what to do. Usually they respond in minutes and do hardware replacements within an hour and small downtimes.
I've also stumbled upon that. However the question remains, how to add more compute nodes? Is there a easy process to do that? How about updating it? Isn't it carless to actually say:
``` You will have sole and unrestricted administration rights to the dedicated hardware with root access. Hetzner Online will not have access to the servers, and will therefore not be able to provide server administration support.
``` ?
I mean if I can't provision it, it's probably problematic to update it.
This isn't quite hitting the spot to me (non-server hardware, you still administer the OS, single location), but it does seem indicative of a potential new wave of low-cost cloud providers.
Someone like Vultr, Linode, or DO is positioned well to deliver an "AWS lite" offering. They all have decent hardware, lots of locations, and a good delivery history. A bit of work to put together ELB/EC2/Lambda/S3 equivalents and a control panel would open up a new market. Especially if they offered low egress pricing.