Leaving the Cloud
In this technical retrospective (2022), @marban shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Pandemic Bubble (2020–2022)
COMMUNITY SCORE 158 points
TOTAL DISCUSSIONS 195 comments
Due to being kind of on the poor side, using AWS, Azure or GCP for personal projects isn't entirely viable for me. I still do utilize some VPS vendors for when I need a decent uptime, but most of the time it's just a GNU/Linux box that I have SSH access to and I configure everything else myself.
Most other things, I run in my homelab, which is just a few old computers repurposed to act as servers for whatever I might need (especially for storing larger amounts of data, e.g. backups), including things like acting as CI/CD nodes, or test environments etc.
I still write "cloud native" software (for example, 12 Factor Apps) because those make the portability of everything pretty well, as well as use something like Docker and some lightweight orchestration on top of it (like Docker Swarm or Hashicorp Nomad) because it's like having the ability to make configuration for what you need (ports, storage, memory/CPU limits, scheduling, restart policies etc.) rather easily and also version it, more easily than Ansible lets me (which I still use for configuration of the server nodes themselves). Basically like multi-node systemd, once could say.
I actually wrote about some of this a year back: https://blog.kronis.dev/articles/how-i-organize-my-docker-deployments-in-2021 ↗
Though since I've also moved over to using Apache as my ingress, for a few reasons: https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-httpd-in-2022 ↗
Either way, it's nice that we still have the ability to base our workloads on open source solutions and learn about all of this stuff on consumer hardware (though certain configurations might be badly supported on choice OSes, like some BSD versions with driver support).
A few days ago I actually published a blog series on making a file and metadata storage solution that's based on open source technologies, which easily handled millions of documents: https://blog.kronis.dev/tutorials/1-4-pidgeot-a-system-for-millions-of-documents-the-data ↗
A student of mine did a rather good thesis on de-clouding. In the process she discovered the term "cloud repatriation" a fair bit of literature on the movement to bring control of data and hardware back 'on-prem'.
She also noted that when you search on these terms the main engines (all run by cloud service providers) return poor results not congruent with the scale of the phenomenon. They are dominated by the opposite message obviously heavily SEO'd up to the top, plus shill pieces pushing cloud services but presented as "critical". Dig deep if you want to find the real scale of the "anti-cloud" issues.
Her main conclusion was very interesting though. That the big issue is not finance, reliability or control - but de-skilling.
As companies move their operations out to the cloud it's not the disappearance of hardware from the premises but the loss of skill-sets. Later they don't even know who to hire or how to write the JD to bring people back in.
A good example was the broadcast industry. Entire branches of that industry doing post-production, colour, transcoding, and whatnot moved it all out to AWS. After price hikes, they wanted to go back to running their own services. But they can't. Because nobody knows about how to set-up and run that any longer - especially specialist things like buffers and transcoders. I mean, try finding a sys-admin who can just do simple tasks like set up and properly configure a mail server these days.
A good definition of The Cloud is just "servers in a datacenter" so the discussion is best framed as Private Cloud (on-prem or co-located servers) vs Public Cloud (AWS, etc) or some combination.
People with expertise in running private clouds and using public cloud services can judge each case pretty easily, by taking into account the organization's technical needs, plans, budget, abilities, etc. And for non-experts, the question is even simpler: use the public clouds if or until it becomes a technical or financial blocker. And in that case, hire an expert.
There's really not a lot of room for controversy. It seems most of the arguments come from people without the relevant expertise, who only know how to develop in public clouds, responding defensively. Like people with little programming experience flaming each other over which language is "best".
There's no single "best" cloud option, since they're just different tools for different things. Sometimes either way would work, in which case it comes down to personal preference, and sometimes there's a clearly better option for a specific organization's needs. Framed correctly, it's a "boring" question just like most technical questions should be.
The HN community 1) likes building and hacking, and 2) has nostalgia for old technology. This makes them feel a lot of affinity toward "doing things themselves" -- server infrastructure, running desktop linux, avoiding dependencies in their code.
The reality, though, is that the steady march of progress encourages us to outsource what we can to people who are better at the thing that is auxiliary to what we do. I don't grow food because I'm bad at growing. I don't repair my car because I'm bad at auto repair.
I do build software, and my company builds a very specific type of software to solve very specific problems. I'm happy to focus on that, because that's how we make money. Other people are much better at building infrastructure than I am, and so I let them do it for me. If there comes a time when the cloud offerings are either worse than what I can build or too expensive, then I trust someone else will come along and fill the gap in the market before it becomes worthwhile for me to do it.
In the early days of cloud providers, their services were both easy-to-use and cost-effective for many growing companies.
Now, cloud providers offer a giant hairball of specialized services, built on top of other specialized services, built on top of yet other specialized services... such that no single person can comprehend all of them, let alone manage them cost-effectively. Moreover, as all these intertwined services become deeply entrenched in IT infrastructure, migrating away from them becomes incredibly disruptive.
The financial incentives of cloud providers are not aligned with those of customers. The more a customer is sucked and trapped inside those giant hairballs, the better for the cloud providers.
To paraphrase the Dark Knight, cloud providers may have been the heroes at first, but they have lived long enough to become the villains.