Hetzner Object Storage
In this technical retrospective (2024), @polyrand shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Post-ZIRP & AI (2023–2026)
COMMUNITY SCORE 248 points
TOTAL DISCUSSIONS 185 comments
Did some benchmarks, not too bad.
``` Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 6 +--------------------+--------------------+------------------+ | ITEM | VALUE | COST | +--------------------+--------------------+------------------+ | upload objects | 207.02 MiB/s | 115.93 ms/object | | download objects | 405.78 MiB/s | 59.14 ms/object | | put small objects | 278.4 objects/s | 21.55 ms/object | | get small objects | 498.5 objects/s | 12.04 ms/object | | list objects | 40977.06 objects/s | 14.64 ms/op | | head objects | 1995.4 objects/s | 3.01 ms/object | | delete objects | 312.1 objects/s | 19.22 ms/object | | change permissions | not support | not support | | change owner/group | not support | not support | | update mtime | not support | not support | +--------------------+--------------------+------------------+ ``` Currently have a project using OVH, which is slower, but than I guess there's currently less load on Hetzner.
``` Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 6 +--------------------+-------------------+------------------+ | ITEM | VALUE | COST | +--------------------+-------------------+------------------+ | upload objects | 80.82 MiB/s | 296.94 ms/object | | download objects | 274.61 MiB/s | 87.40 ms/object | | put small objects | 31.6 objects/s | 189.98 ms/object | | get small objects | 145.1 objects/s | 41.36 ms/object | | list objects | 6293.39 objects/s | 95.34 ms/op | | head objects | 177.7 objects/s | 33.76 ms/object | | delete objects | 136.6 objects/s | 43.93 ms/object | | change permissions | not support | not support | | change owner/group | not support | not support | | update mtime | not support | not support | +--------------------+-------------------+------------------+ ```
A while ago I needed something managed, very cheap and S3 compatible and I went with Contabo Object Storage: https://contabo.com/en/object-storage/ ↗
Admittedly I got my money's worth, for about 3 EUR for the very basic plan of 250 GB storage, I get upload speeds of about 6-9 MB/s and download speeds of about 25 MB/s, which is okay for what I'm trying to do. That said, it doesn't seem like there's a way to create additional users with access to only specific buckets not all of them for the same service and the overall offering does feel a bit jank (e.g. when you don't have an active service, you can't log into Contabo at all and while their Object Storage site is nice, the regular VPS one feels functional but dated).
What I'm saying is that when it comes to budget options, Hetzner will probably do great, since they have a good track record and it's not like there are that many other alternatives out there!
Of course, I could have also gone with just self-hosting with MinIO, Garage or SeaweedFS, as long as the VPS that I'd get would also have enough storage. It's nice that there are self-hosted options, too, I almost dread when I see bespoke object storage solutions at work, either developed before S3 was a thing or because the devs just didn't know or had a case of NIH, so I have to look at how a bunch of blobs are passed through servlets and serialized/deserialized, as well as deal with custom metadata and permission mechanisms.
Since we're talking object storage, a question for the collective brain: are there any object storage solutions, cloud or on-prem, which support any sort of "operator pushdown"?
By "operator pushdown", i mean any ability to filter or map over the contents of the object on the server side in some way, sending only the results over the network to the client.
For example, say you have a huge CSV file of customer orders in a bucket. You might want to find the timestamp of all the orders which included a particular product. If all you can do is stream the whole file, then you need to do that, just to pick out a few timestamps. But you could imagine a kind of request where you say "only give me lines where the product ID is P01234, and only send the timestamp column". Perhaps you would express that as a pair of regular expressions, or a sed program, or a Lua script, or maybe the server would understand CSV and let you write something a bit like SQL. There are all sorts of ways it could be done. Providing a fully general way might be tricky, but it wouldn't need to be fully general to be useful.
I appreciate that if you want to do this sort of access frequently, you should probably be using a database, not object storage. But it seems like a very useful feature to layer on top of object storage, and one that feels like it should be fairly cheap to execute - the server has to do a small extra amount of computation, but then needs a lto less network bandwidth.
So, I don't want to sound like a conspiracy theorist, but this announcement has extremely suspicious timing.
You see, I host a cloud storage service on Hetzner. I have 12 PETABYTES of user data stored there. The data is spread over 120 of their SX type storage servers. Last week Hetzner sent me an email saying they were closing my account for an unknown reason. I have repeatedly been asking about the reason of this sudden closure, but they won't tell me. Meanwhile I have a huge problem, because I need to move 12 PB to a different hosting provider, and they only gave me until the end of November. That's an almost impossible deadline for setting up such a large storage cluster. Especially considering that Hetzner's 1 Gbps port speed makes it impossible to transfer the data in less than three weeks.
Don't trust Hetzner, guys. They screwed me over real bad here, and it seems like they will be taking the hardware that I have been renting there for 10 years for themselves now. This is incredibly scummy behaviour.
As someone who has managed many public Ceph clusters at Linode (now Akamai) since 2016 for both block and object storage, I wish the Hetzner engineers good luck!
There are a _lot_ of challenges in keeping the clusters secure, reliable, and performant. Make sure you have systems or tools in place to prevent abuse. Be aware of the little nuances of Ceph, like what time lifecycle policies kick off, or when dynamic bucket resharding will kick in (and block client writes!).
If possible, conduct extensive failure testing in a lab environment under simulated load to see how your clusters will really behave when it eventually happens. Triple check all of your tunables and your pool configuration. Some things like erasure coding profiles are set in stone, and once you have customer data on your clusters, there is no turning back.