RethinkDB: why we failed (2017)

As someone using RethinkDB in production for the last 6 years or so: I am really disappointed by how software development world is dominated by fashions and fads.

RethinkDB was done really, really well. It is one of the very few distributed databases that went through Jepsen relatively unscathed and delivered on promises made. Development was done in the public, questions were asked through StackOverflow. You interacted with competent, skillful and experienced developers.

And yet MongoDB was the latest fashion fad, end even though it did NOT deliver on the promises, it was the hot-database-du-jour that kids used.

The article is mostly about the business model, and I also always thought they would have a hard time making money on the database. But I think they would have had a much better shot if it became more popular. It didn't, which accelerated the company's demise.

EDIT: I just realized that I wrote about RethinkDB in the past tense, even though it very much exists as I write it. In fact I run my business on it. But because it fell out of favor, when it was open-sourced, it failed to pick up momentum, and it now seems unlikely that it will.

Which is why I'm working on switching to FoundationDB, and I'm slightly worried that it will suffer the same fate: it is excellent technically (the best transactional guarantees in a distributed database you can get), but difficult to understand and not very user-friendly. It's not the "node.js database for everyone". The only reason I'm considering it is because Apple uses and develops it, which gives me hope for longer-term maintenance.

Going back to fashions — you can have a product which excels technically, but if it's out of fashion, it might as well not exist.

I think we would all be better off if we stopped trying to always pick The One True Database, The One True Programming Language, etc — and instead accepted that there might be multiple tools, each specialized for certain kinds of tasks.

I had such high hopes for RethinkDB as a free and open source alternative to Firebase. FB was a great tool, but was never the same after getting purchased by Google (I quit following its progress shortly thereafter).

This article has less to do with databases or open source software than it does with the fundamentally misaligned incentives of egalitarianism and capitalism.

I wish there was a social open source license that said "you can use this indefinitely as long as you pay us something". The price would be up to the user but the generally accepted polite minimum would be at least a penny ($0.01).

Then businesses could publicly display their level of support (both initially and yearly) for the software they use, to attract customers. They could also be audited by the IRS, so a lack of patronage could correlate to a lack of equality and maybe even reveal corruption and other malfeasance. At the very least it would reveal a lack of internal controls.

This could work kind of like UBI for open source. And I'm definitely not the only one who has ever thought about this. But there is just so much free capital floating around right now that it's a great time to be thinking about how to reform the paradigms that we all depend on.

So much of this is familiar. I was at two database startups.

The first one, an object-oriented database company (last 80s, early 90s) suffered from an extreme case of misperceived markets, due to the occasionally intentional blurring of the lines between a "database system" (which means SQL to nearly everyone, and we didn't to SQL), and a persistent storage class added to C/C++ (and later Smalltalk and Java).

The second one (mid 2000s) was a neat physical storage trick, which should have been a feature of a database system, not an excuse for building a new one. My bad for missing this, but I really, really liked the physical storage ideas, so I joined.

Building and selling new database technology is extremely difficult. There are successful, huge, entrenched companies. There is, therefore, no good reason for anyone to gamble on your new technology. Your only chance is to convince the architecture astronauts at some prospect that they just have to have your product, but the odds of such a decision sticking all the way through the delivery of their product is extremely low, no matter how good your technology is.

This is akin to everyone’s dream to start a cozy, atmospheric coffee shop. These types of coffee bars fold very quickly or at best give founders years of servitude below minimum wage. The reason is that the type of behaviour this kind of establishment encourages - lounging, book reading, laptop work is exactly the opposite of the quick serve model that is conducive to high revenue. The customer loves this model, the owner hates it.

Analogous to it, the nosql DB market is the exact wrong market to enter. Large companies willing to pay big bucks for enterprise features will pay established vendors for the stodgy but battle tested stuff like Oracle or DB2. The hipster startup market pays nobody for anything as there is a myriad of free choices in every common flavour and the few that will pay will mostly do so to purchase managed hosting. And that’s only if their PoC built on your new and untrusted database ever makes it to production. And then it’s probably your cheapest tier. Don’t be selling to paupers!

What’s with all of this seemingly whitewashing of history? Slava himself described why it failed. I mean as founder you have to be laser focused. Yet he was too busy trying to win over his “hot new employee” than focusing on the database.

There is an archive of his deleted tweets here where he describes it: https://gist.github.com/travisbrown/059310042193a2e143408b05bdc2278d ↗

And then there was an article that seems to be gone from the web (https://web.archive.org/web/20201130215752/https://threader.app/thread/1124414461014462464 ↗). It stands to reason this huge distraction played a role in the mismanagement of the database and why it ultimately didn’t succeed.