We cut our Mongo DB costs by 90% by moving to Hetzner

Here's how we managed to cut our costs by 90%

You could cut your MongoDB costs by 100% by not using it ;)

without sacrificing performance or reliability.

You're using a single server in a single datacenter. MongoDB Atlas is deployed to VMs on 2-3 AZs. You don't have close to the same reliability. (I'm also curious why their M40 instance costs $1000, when the Pricing Calculator (https://www.mongodb.com/pricing ↗) says M40 is $760/month? Was it the extra storage?)

We're building Prosopo to be resilient to outages, such as the recent massive AWS outage, so we use many different cloud providers

This means you're going to have multiple outages, AND incur more cross-internet costs. How does going to Hetzner make you more resilient to outages? You have one server in one datacenter. Intelligent, robust design at one provider (like AWS) is way more resilient, and intra-zone transfer is cheaper than going out to the cloud ($0.02/GB vs $0.08/GB). You do not have a centralized or single point of failure design with AWS. They're not dummies; plenty of their services are operated independently per region. But they do expect you to use their infrastructure intelligently to avoid creating a single point of failure. (For example, during the AWS outage, my company was in us-east-1, and we never had any issues, because we didn't depend on calling AWS APIs to continue operating. Things already running continue to run.)

I get it; these "we cut bare costs by moving away from the cloud" posts are catnip for HN. But they usually don't make sense. There's only a few circumstances where you really have to transfer out a lot of traffic, or need very large storage, where cloud pricing is just too much of a premium. The whole point of using the cloud is to use it as a competitive advantage. Giving yourself an extra role (sysadmin) in addition to your day job (developer, data scientist, etc) and more maintenance tasks (installing, upgrading, patching, troubleshooting, getting on-call, etc) with lower reliability and fewer services, isn't an advantage.

What most of the commenters here are missing is the reality that not every system, function, or business needs the sort of uptime that AWS offers - and that's fine. It's something a lot of newer entrants into the technology field fail to grasp, because they've never had to actually deal with an outage before - or a time when the internet itself was ephemeral and temporary, available only as long as your connection remained active.

The number one thing people poo-pooing these "We saved $XXX by getting off public cloud" posts is that each business has different calculus for its risk tolerances, business needs, and opportunity costs. Once a function reaches some form of stability or homeostasis, then hosting it in the public cloud can become a net liability rather than a net asset. Being able to make those decisions impartially is what separates the genuinely good talent from those who conflate TC with wisdom.

Even when public cloud is the right decision, using managed services increasingly isn't. MongoDB Atlas is a managed service with a corresponding price tag to match. Running it in a VPS like Hetzner may shift some of the maintenance and support tasks onto your team, but let's be real - modern databases are designed to be bulletproof, and huge companies operated just fine with a single database instance on bare metal for decades, even with the odd downtime along the way. We ran a MongoDB CE database at a PriorCo on a single VM in a single datacenter for nearly a decade, and it underpinned a substantial chunk of our operations - operations we could do by hand, if needed, during downtime or outages (that never happened). We eventually moved it to AWS DocumentDB not out of cost-savings or necessity, but because a higher-up demanded we do so.

If anything, the visceral rebuke of anyone daring to move off public cloud feels very reminiscent of my own collegial douchebagginess in the 2000s, loudly mocking Linux stans and proclaiming closed source (Microsoft) would run the planet. Past-me was a douchebag then, and the same applies to the AWS-stans of today.

I wonder how many companies are running databases on non-encrypted storage on Hetzner.

Their bare-metal servers don't have storage encryption by default, and I don't know for sure about the VM hosts, as I don't have access, but Hetzner never claims that it is encrypted at rest. And there is no mention of storage encryption in their data protection agreement. https://www.hetzner.com/AV/DPA_en.pdf ↗

Also, their data privacy FAQ mentions "you as the customer are responsible for both the data that is stored on your rented server and for the encryption of that data." https://docs.hetzner.com/general/general-terms-and-conditions/data-privacy-faq/ ↗

I would recommend, just in case, to set up LUKS on your server. You will find many guides for Hetzner.

If you don't do that, seeing your data in the wild is a real scenario. A few years ago, a Youtuber bought some used hard-drives in the hope to recover data to illustrate the risks of not erasing a hard-drive correctly. He eventually bought a hard-drive containing non-encrypted VM disks from Scaleway, a Hetzner competitor. My guess is that some hard drives disappeared before destruction after being decommissioned. Some customers got their shitty source code exposed on a 1.4M views video. Here is the first one: https://www.youtube.com/watch?v=vt8PyQ2PGxI ↗

So, use LUKS.

The more keen eyed among you will have noticed the huge cost associated with data transfer over the internet - its as much as the servers! We're building Prosopo to be resilient to outages, such as the recent massive AWS outage, so we use many different cloud providers.

I mean, you're connecting to your primary database potentially on another continent? I imagine your costs will be high, but even worse, your performance will be abysmal.

When you migrate to a self-hosted solution, you're taking on more responsibility for managing your database. You need to make sure it is secure, backed up, monitored, and can be recreated in case of failure or the need for extra servers arises.

...for a small amount of pain you can save a lot of money!

I wouldn't call any of that "a small amount of pain." To save $3,000/month you've now required yourself to become experts in a domain that maybe is out of your depth. So whatever cost saved is now tech debt and potentially having to hire someone else to manage your homemade solution for you.

However, I self-host, and applaud other self-hosters. But sometimes it really has to make business sense for your team.

I love MongoDB's query language (JS/Node.js developer so the syntax fits my mental model well), but running a production replica set without spending tons of cash is a nightmare. Doubly so if you have any unoptimized queries (it's easy to trick yourself into thinking throwing more hardware at the problem will help). Lord help you if you use a hosted/managed service.

Just fixed a bug on my MongoDB instance last night that, due to a config error w/ self-signed certs (the hostname in the replica set config has to match the CN on the cert), that caused MongoDB to rocket to 400% CPU utilization (3x, 8GB, 4VCPU dedicated boxes on DO) due to a weird election loop in the replica set process. Fixing that and adding a few missing indexes brought it down to ~12% on average. Simple mistakes, sure, but the real-world cost of those mistakes is brutal.