Hetzner Servers Compromised
In this technical retrospective (2013), @martindale shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Web 2.0 Genesis (2007–2013)
COMMUNITY SCORE 180 points
TOTAL DISCUSSIONS 130 comments
Full text of the email sent to cutomers:
Dear Client
At the end of last week, Hetzner technicians discovered a "backdoor" in one of our internal monitoring systems (Nagios).
An investigation was launched immediately and showed that the administration interface for dedicated root servers (Robot) had also been affected. Current findings would suggest that fragments of our client database had been copied externally.
As a result, we currently have to consider the client data stored in our Robot as compromised.
To our knowledge, the malicious program that we have discovered is as yet unknown and has never appeared before.
The malicious code used in the "backdoor" exclusively infects the RAM. First analysis suggests that the malicious code directly infiltrates running Apache and sshd processes. Here, the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected.
The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code.
We have commissioned an external security company with a detailed analysis of the incident to support our in-house administrators. At this stage, analysis of the incident has not yet been completed.
The access passwords for your Robot client account are stored in our database as Hash (SHA256) with salt. As a precaution, we recommend that you change your client passwords in the Robot.
With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.
Hetzner technicians are permanently working on localising and preventing possible security vulnerabilities as well as ensuring that our systems and infrastructure are kept as safe as possible. Data security is a very high priority for us. To expedite clarification further, we have reported this incident to the data security authority concerned.
Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in regard to this incident.
Naturally, we shall inform you of new developments immediately.
We very much regret this incident and thank you for your understanding and trust in us.
A special FAQs page has been set up at http://wiki.hetzner.de/index.php/Security_Issue/en ↗ to assist you with further enquiries.
Kind regards
Martin Hetzner
Several events with Linode, now Hetzner. These are relatively "premier," high-quality hosting companies, you can count on thousands and thousands of companies to pay even less attention.
Yet every time, the discussion is only about one specific company, without seeing any broader pattern.
When are we ever going to draw the conclusion that popular hosting companies (and, actually related, facilities like RubyGems) are especially attractive targets, and that the approach of waiting for an exploit and then shaming the targeted company is not an effective way of getting better security?
The info in the mail regarding the safety of credit card info contradicts with the linked FAQ.
FAQ: Bank details are encrypted (two-way) in the database. However, it cannot be excluded that the attacker/s have also been able to obtain access to the key.
Mail: With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.
One of the differences between servers in data centers and desktops is that servers don't reboot a lot. So the bad guys can build an in memory system that doesn't change anything on disk (which avoids the configuration management system from flagging it) and if it cloaks itself as a normally long lived process then active monitors might be fooled as well. It then runs, effectively undetected, until the server reboots which can be months (or even years).
Tools to protect against those threats are going to have start taking into account process activity and footprint.
The lesson to take from this is you can't rely on someone else providing security for you. You need to make sure you "layer" your security. If your database is compromised make sure you are salting your passwords or at least using a hash that is designed to be resistant against cracking.
This doesn't protect against someone who has root access to your box and manages to capture passwords in the clear as they are being transferred but it does limit the surface of a breach.