How we built ngrok's data platform
In this technical retrospective (2024), @samber shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Post-ZIRP & AI (2023–2026)
COMMUNITY SCORE 164 points
TOTAL DISCUSSIONS 40 comments
I found the technical details really interesting, but I think this gem applies more broadly:
I have found this to be so true. SWE vs DE is one division where this applies, and I think it also applies for SWE vs SRE (if you have those in your company), data scientists, "analysts", basically anyone who is in a technical role should ideally know what kinds of problems other teams work on and what kinds of tooling they use to address those problems so that you can cross-pollinate.
15k/s event rate and 650GB volume / day is massive. Of course that's confidential, but I'd guess they are below 10k concurrent connections. So they are recording 1.5 event's / second / user. Does every packet need discrete & real-time telemetry? I've seen games with millions of active users only hit 30k concurrents and this is a developer tool.
Most events can be aggregated over time with a statistic (count, avg, max, etc). Even discrete events can be aggregated with a 5 min latency. That should reduce their event volume by 90% . Every layer in that diagram is CPU wasted on encode-decode that costs money.
The paragraph on integrity violation queries was helpful -- it would be good to understand more of the query and latency requirements.
The article is a great technical overview, but it's also helpful to discuss whether this system is a viable business investment. Sure they are making high margins, but why burn good cash on something like this?
Blimey, that is a lot of moving parts.
Our data team currently has something similar and its costs are astronomical.
On the other hand our internal platform metrics are fired at BigQuery [1] and then we used scheduled queries that run daily (looking at the -24 hours) that aggregate/export to parquet. And it's cheap as chips. From there it's just a flat file that is stored on GCS that can be pulled for analysis.
Do you have more thoughts on Preset/Superset? We looked at both (slightly leaning towards cloud hosted as we want to move away from on-prem) - but ended up going with Metabase.
[1] https://cloud.google.com/bigquery/docs/write-api ↗
Don't worry, your sensitive data isn't handled by our platform, we ship it to a third-party instead. This is for your protection!
(I have no idea if Clickhouse is actually a third party, it sounds like one though?)
Kudos to the author who is responsible for the whole stack. A lot of effort goes into ingesting data into Iceberg tables to be queried via AWS Athena.
But I think it's great that analytics and data transformation is distributed, so developers also are somewhat responsible for correct analytical numbers.
In most companies there is strong split between building product and maintaining analytics for the product, which leads to all sort of inefficiencies and errors.