"Hetzner decided to cancel our account and terminate all servers"

The biggest issue I have had with Hetzner was with a dedicated server. I was constantly (3 times or more a week) getting abuse messages about my MAC address not being correct:

"""" We have detected that your server is using different MAC addresses from those allowed by your Robot account.

Please take all necessary measures to avoid this in the future and to solve the issue. We also request that you send a short response to us. This response should contain information about how this could have happened and what you intend to do about it. In the event that the following steps are not completed successfully, your server can be locked at any time after DATEHERE.

How to proceed: - Solve the issue - Please note, in case you have fixed the problem, please wait at least 10 minutes before rechecking: https://abuse.hetzner.com/retries/?token=TOKENHERE ↗ - After successfully testing that the issue is resolved, send us a statement by using the following link: https://abuse.hetzner.com/statements/?token=TOKENHERE ↗

Please visit our FAQ here, if you are unsure how to proceed: https://docs.hetzner.com/robot/dedicated-server/faq/error-faq/#mac-errors ↗ """

I was just using standard Docker to host a web app. No proxmox or KVM of any sort. I would just wait the 10 minutes, click their link https://abuse.hetzner.com/retries/?token=TOKENHERE ↗, which would retry and would come back fine and my response would be "I changed nothing and the retry came back solved. I've done tcpdumps over a weeks time to see if any MAC addresses leak from the OS and none have while a similar ticket like this gets opened every couple days." The ticket would close shortly after I submitted.

I inquired to them at least twice about this and they just kept telling me I was leaking a MAC address that I wasn't allowed to even when I had proof of tcpdumps over a week time period. I found someone else who had this issue with them (most issues around this that I found were people hosting Proxmox) and they had Hetzner replace the NIC and it fixed the issue. Well, Hetzner wouldn't replace my NIC because "it was working" even though I referenced these abuse tickets. I ended up getting another dedicated server, migrated my app over there, and I haven't had issues since.

Their support is seriously not very good. Since that experience, I have had backups elsewhere and test restoring those backups regularly. The price to performance I get from them is unbeatable and like I said, I haven't had issues since getting a new machine. But, I'm definitely cautious and don't exactly trust things to not go sideways even though it's been 2 years since that experience.

For anyone else who needs to hear this,

Hi,

I don’t have a mastodon to reply directly to you.

But i have had some issues with content being taken down by VPS providers as well.

What I’ve found works well is to use a VPS provider that the public is unaware of. And for some time I had used OVH based on the unlimited bandwidth and the reasoning that Wikipedia and Julian assange (who have far more enemies than I ever will) were using OVH.

I don’t know if that’s true any more because I subsequently moved my content to ENS and IPFS.

Anyway regardless of where your content is actually hosted or lives,

What I had done was turn my “real” servers into content origins , which were concealed form the rest of the world and lock it down in the firewall so it could only be reached by disposable squid proxy servers with a 10-liner config file

Then I pointed DNS , cloud flare etc at the squid nodes

And couldn’t care less if they were taken down.

Because I could deploy new ones in minutes elsewhere.

I didn’t have “bad content”, just ruthless business competition that kept coming at me like Tonya Harding.

And I’m sharing because your content didn’t seem too offensive either.

In the front end VPS nodes you’d just put the real address of your content as the remote origin.

And then nobody but you will ever know where it is.

Then generally your hosting company shouldn’t be aware of what it is either unless they’re snooping around in your files, and if they are, hell with them too.

You’re welcome to pass this along as a remark on avoiding censorship, or keep it to yourslelf as proprietary information I don’t mind. Let me know if you want or need an example squid conf. It’s seriously 10 lines at most and many examples found on google.

Three possibilities come to mind:

1) There is some fundamental data aspect Kiwix hasn't mentioned (or is entirely unaware of). I.e. CP or some other super illegal stuff.

2) Hetzner is profoundly incompetent, deleted production servers by accident, and the "But we sent you an email!" thing is a lie to cover up the mistake.

3) There is some kind of interaction that happened prior to this that we aren't privy to. Perhaps a series of late bills, legal threats, or some other inter-personal issue.

Predicted outcome:

I either expect Kiwix to get a knock by federal/national authorities. Or the more likely outcome in my opinion: some frustratingly vague statement by Hetzner PR about its customers being "mistaken" in regards as to why data-go-poof.

I mean seriously, let us assume it's something illegal: Sure, fine, whatever. Wouldn't it make more sense for that material to not be deleted, so whoever the guilty party is would be arrested for/prosecuted by it? Deleting the servers would be like police being informed about a murder weapon and asking the tipster to destroy the weapon before an arrest is even made. It doesn't make any sense to me. Surely if some bad thing were discovered, there would be some method to encrypt/restrict illicit material without destroying it.

Either bad blood, or unpaid bills, or simple incompetence seems like the most likely culprits to me.

Heh, this happened to me the other day. I had hooked it up to paypal and didn't realize it wasn't set up to autopay, so I had an outstanding balance of $8 for about a week and they nuked everything. It was just for a hobby project so it was no big deal and I'll provision a new server with them, but I'm not sure I'd use them for a serious project, even though their prices are good. Granted, if it was for a serious project, I would have spent more time and care setting stuff up.

The funny bit was I paid the invoice, and then my account remained suspended. When support finally got back to me a few days later, they said (and I quote)

``` Dear Client We want to give you one last chance as a gesture of goodwill, so we revoked the cancellation for you. Kind regards ``` which made my account accessible again. You'd think they'd be a little lenient for new accounts where the debit is less than $10, but I guess not.

Most of the times you hear people complaining about Hetzner shutting down someone's servers, it's because they were hosting content going against their ToC or similar.

But this seems to be about Kiwix (which in short is "offline Wikipedia" in various ways) and doesn't seem to be about questionable content in any way.

Eventually I guess we'll get Hetzner's perspective on this, as they tend to start writing publicly about issues once the other side starts writing publicly about it as well.

Personally I've been a happy user of Hetzner for many years, with no issues that weren't my own doing. But reading about people having their servers deleted in the middle of the night on a Sunday (Berlin time) and all data wiped immediately, with no recurse, does sound a bit aggressive. Luckily it seems like both me and Kiwix has mirrors for the data we care about.