How We Built the Software that Processes Billions in Payments
In this technical retrospective (2011), @drewolson shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Web 2.0 Genesis (2007–2013)
COMMUNITY SCORE 85 points
TOTAL DISCUSSIONS 38 comments
Testing: testing is at the forefront of our development philosophy. We never need to check our code coverage to know that it's at 100%: with disciplined TDD, no line of code will be written without a test.
Bravo. In my experience that can be overkill, but with finance, I agree: why risk it. TDD everything.
We don't have a QA team.
WTF?
That might be terrifying when you consider the type of software that we're building, but we're confident that our automated testing is thorough and will catch any regression bugs.
Are regression bugs the only kind of bugs?
We use continuous integration to test every version of every client library against our gateway.
What happens when someone uses your client library in a way you didn't anticipate?
What happens when johnny-botnet hits your API directly without using the client library?
I spent several years developing games, with QA teams that outnumbered the developers. The QA team did not just play the levels through and say "it works!". Sure, they did that for the first hour. Then they'd start doing all those things that they thought someone might try (e.g. in a fit of boredom, or for a laugh). After that they'd just try breaking stuff. What a lot of bugs they would find!
As I write I find it hard to believe that I, a game developer, is having to explain the importance of QA to a financial company.
How do you guys maintain an ever growing code base?
It seems to me that the practices you have--TDD with full code coverage and regression tests, pair programming everything, many client libraries--would lead to a massive amount of code that needs to be maintained.
In my experience I've found that TDD often leads to a huge amount of code that needs to be rewritten if you do a major change to your codebase. If you have a new pattern you want to implement it may require rewriting dozens or more tests. Also, when I'm working alone I constantly go back and clean up functions and what not to make them shorter and clearer. I've found I do that less while pairing because you want to keep moving forward.
My opinion is probably just biased by the fact that I've done relatively little pair programming and TDD. I'm sure the more you do the better you get.
But I'm curious if you could take some about the size of your code base and how these practices effect that?
...we are able to perform all our maintenance without downtime. We can deploy new versions of our software, make database schema changes, or even rotate our primary database server, all without failing to respond to a single request. We can accomplish this because we gave ourselves the ability suspend our traffic. To make this happen, we built a custom HTTP server and application dispatching infrastructure around Python's Tornado and Redis.
Why is it necessary to suspend traffic to make these kinds of changes? Just curious.
And that is how you write a great job advert.
More specifically, it gets across:
- the company culture
- the development practices/methodologies in place
- what's exciting about the work/company
- what technologies they like (and the fact they use a variety of technologies)
- the fact that they get that typical job adverts suck
... all while looking like a regular article.
Great work.
Different strokes for different folks i guess. I know a lot of people will be fans of this, but
No thanks, if I'm the developer. And if i wouldn't do it, why would i make my employees.