Why we are not leaving the cloud
In this technical retrospective (2017), @happy-go-lucky shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Cloud & Mobile Era (2014–2019)
COMMUNITY SCORE 350 points
TOTAL DISCUSSIONS 212 comments
Why was hybrid not considered? With providers like megaport providing really inexpensive direct connect, this is almost a no brainer.
This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this.
Physical environments DO work well, but they do require experience to run them. I have used AWS for as long as its been around, but nothing beats physical environments for "known workloads", as long as you run them EXACTLY like you would run a cloud environment. This is why the hybrid approach is such a great thing. Run what you know well in physical and reap great savings, run what you dont know well cloud ( as well as take advantage of the analytics products), and reap the speed ;)
- This means thinking of physical servers are individual units. - This means redundancy at every level. - This means architecting for failure ( servers and switches do gie ). - This means no shared storage for performance critical parts ( shared storage as below the OS level ). - This means objects stores/sharding/etc as your storage layer. - This means real engineering - And this means exactly the same whether you're physical or not.
I've managed environments of 50 vm's and environments of 50000 physical servers. The methodology is always the same.
and yes, this means you can save some monthly cost, and apply it to staff, that can do a lot more than just maintain this infrastructure.
PS: for those that think that showing up to a datacenter is required, you're doing it wrong. Pretty much any datacenter has hot-hands service, and with the right redundancy, hardware replacement is something you can do at a slower pace.
PPS: im sure someone will nitpick some of my points. The reality is, there is real money savings here. For example, Snapchat spends MORE in cloud infrastructure in 2016/2017 per year, than ALL OF GOOGLE did in 2012... think about that for a second... even netflix runs openconnect to push bits..
So everyone says the cloud is the future. I get it. But is this the truth, or what all the tech giants want people to believe? Don your foil hats for a moment and listen to me.
All the original internet companies run their own hardware. They rent out excess production capacity to us peons in the form of cloud services.
These companies that all run their own hardware exclusively are telling everyone that it's stupid to run your own hardware... Why are we listening?
Look to the newer tech giants as a prequel of what's to come. Netflix for example, is completely at the mercy of Amazon. They might as well be "Netflix brought to you by Amazon". Their edge is in software alone, nothing that causes a huge barrier to entry like custom hardware. This makes them much easier to dethrone. Hilariously, they rent all their hardware from a direct competitor who has access to all their software secrets. Does anyone else see something wrong with this?
The cloud as a money saving venture is and always has been a damn lie.
It's the same tactic as when automotive companies paid off local governments to destroy Americas public transport many years ago. All the big tech companies have their hands in the cookie jar besides Facebook, who remains mostly silent but runs their own hardware as well.
The major tech companies have every incentive to make you think running your own servers is nigh impossible. Don't drink the fucking koolaid. If the industry continues to consolidate rapidly AmaGooSoft will be the ONLY places you can have web servers in ten years
As much as I like gitlab, that decision seems weird. I'd expect a business to come up with numbers for different scenarios, not just quotes (even if those weren't the only deciding factor).
What I mean is, instead of writing about 8TB disks vs 2TB disks, get hard numbers for those and calculate costs for different versions to the costs in the cloud.
Instead of saying that engineers are expensive, look at market data to determine how much you have to pay to hire these people.
Instead of writing about the potential of full cabinets, get quotes from data center providers of what they can provide and how they would deal with this. I'm sure Equinix has solutions for that kind of problem.
And in addition to that, it seems that managed dedicated wasn't considered at all. Why hire someone to swap disks for you? Why not just renting capacity for servers and have the data center provider taking care of hardware issues? They operate at bigger scale and can do this with little overhead costs. You're still much cheaper than the cloud but don't need to drive somewhere to swap a server (you just switch to a spare one until the data center fixed the problem). With Gitlab's scale, it shouldn't be too hard to get a decent bespoke quote from companies like OVH.
Every time I see a post from Gitlab, I worry for them. I haven't used gitlab but have checked it out a few times. On the other hand I use github regularly. But this isn't about github vs gitlab.
Instead this is about gitlab's definition or rather level of transparency and something they seem to be proud of and are even being appreciated very often here at HN. I feel that gitlab have taken the transparency thing to an extreme, to the extent that they even named the engineer that caused a recent downtime. I don't think it matters that the engineer didnt object to it. Then there are posts about why they are choosing some UI framework and why they are moving away from cloud and now this. It's fine to be transparent but in a professional world it's also important to be private about certain things. Trying to please people by being transparent for the sake of it will take a toll on the team eventually. It's much more effective to just get things done instead of constantly being transparent to the external crowd, about every single detail
I love all the comments trying to explain that managing your own hardware, storage, network and backup cannot take that much of your time.
Not only it does take a huge amount of time, but it takes so much time that it require multiple dedicated highly-skilled people.
Last but not least, the last article about Gitlab was about a major outage where they had as much as 6 out of 6 backups unusable! The fact that they still exist today is to the sheer luck that there was an unexpected copy made manually before the disaster.
That speaks at length on how that 100 people company is struggling to have the resources and the qualifications it needs to execute (what startup isn't?). Let's not send them on a suicide mission to manage their own hardware on top of everything they already have to achieve. ;)