RethinkDB Postmortem

Slava (coffeemug) is very brave to introspect publicly and whenever I interacted with him, behaved with greatest integrity and kindness. Unfortunately, his hypothesis here that devtools are a tough market does not ring true. Making a business is always tough, making one that does something new is even tougher. There are many tools and database companies doing extraordinarily well, from Atlassian to Cloudera to Docker to Elastic. The difficulties that RethinkDB faced, as it pivoted from closed source MySQL DB backend, to open source independent database with a very cool update mechanism, were in my analysis much more mundane. Simply, RethinkDB was not easy. Here is a flavor of their thinking, when they rejected a REST HTTP interface: "The RethinkDB query language is so simple, expressive, and pleasant, that trying to make it available over HTTP would be quite difficult and a disservice to users."

Developers have a big advantage when making devtools in that they are their own target market. They are, but only at the start of the project before they become embroiled in it. Ironically this feeling of making it for oneself leads to a mistaken urge to thrust complexity onto the target. As an infra engineer myself, this is something I always try to keep in mind -- I worked on a modelling language at a large company and I credit its success to its absolutely straightforward and accessible syntax, lack of dependencies and straightforward integration into workflows. The temptation is always to make sure users are aware of the choices, force them out of bad habits and make them consider the tradeoffs (this Strange Loop talk from Peter Alvaro addresses the social treatment of users https://www.youtube.com/watch?v=R2Aa4PivG0g ↗ ). For RethinkDB, it was wrong to try to force people to learn a new language to use a DB, when there are just tons of interfaces already, and the new language has incremental benefits over say SQL.

However elegant the design of software, the deployment that is the largest or most demanding for it will always find problems. There's no point seriously trying to claim great reliability or quality for a nascent project. It's unbelievable. Instead you need to focus on getting it used in some friendly project that is willing to pay the cost of these teething problems for some particular feature. Once battle tested, reasonable tech leads can choose to use it, comfortable that they won't be the biggest deployment (e.g. git bootstrapped by its use in the Linux kernel). Alternatively, and less scrupulously, you can try to bootstrap by providing a very convenient experience at the start of a simple project for junior engineers - there are many NoSQL DBs that promise the world and are incredibly easy to set up (see the https://aphyr.com/tags/jepsen ↗ Jepsen blog for many sad examples, RethinkDB proudly did well in this test).

RethinkDB refused to make the market fit compromises necessary. Market fit is about timing. Don't read the Economist. Ship early and keep your users on board through whatever engineering compromise necessary.

Fascinating read, albeit a sad one: I love RethinkDB, and use it personally every day in our particular production deployment.[1] But -- and to Slava's point -- we didn't/don't/wouldn't pay for it, and in that regard, we were part of the problem. That said, the constraint on that particular problem was that it had to be open source, and if it couldn't have been RethinkDB, it would have been some other open source database -- proprietary software is a non-option for that use case (or indeed, for any of our use cases).

So I feel Slava's pain (and feel somewhat culpable as a user of RethinkDB), but I don't entirely agree with the analysis. Yes, infrastructure is a brutal space because it has been so thoroughly disrupted by open source -- but the dominance of AWS shows that open source can absolutely be monetized (just that it must be monetized as a service). Indeed, right now, I would love to deploy a RethinkDB-based service -- and if the company still existed, we (and by "we" I mean "we Samsung", not "we Joyent"[2]) would potentially be in a serious business conversation about what a supported, large-scale deployment would look like. Actually, that's not entirely true, and it leads to me to one thing that Slava didn't mention. While it kills me to bring it up, it does represent the single greatest impediment I have personally found to deep, mainstream, enterprise adoption of RethinkDB: its use of the AGPL. Anyone who has been part of a serious due diligence knows that the GPL alone is toxic to much of the enterprise world -- and the GPL is a buttoned-down corporate stooge compared to the virulent, ill-defined, never-tested AGPL. The AGPL is a complete non-starter for any purpose, and while I appreciate why the AGPL was selected by RethinkDB (and very much appreciate the explicit riders that RethinkDB put on it to make clear what it did and didn't apply to), the reality is that no one -- absolutely no one -- has built a business on AGPL software and it seems vanishingly unlikely that anyone ever will.

I still use RethinkDB and still intend to -- and I still harbor hope (crazily?) that whatever entity that owns the IP will see fit to relicense it to something that is palatable to the kind of people who care about their data. I actually believe that a business can be built on the technology -- which, to Slava's points, I have found to be delightfully sound in both design and implementation. And that RethinkDB is open source means -- for us, anyway -- it will continue to live on, despite the unfortunate demise of its corporate vessel (AGPL or no). That said, I would love to have a lot more company -- here's hoping that we see RethinkDB relicensed and a thriving business behind it!

[1] https://github.com/joyent/manta-thoth ↗

[2] https://www.joyent.com/blog/samsung-acquires-joyent-a-ctos-perspective ↗

As someone who has worked at a bootstrapped B2B FLOSS company in the ERP/database domain I think taking/relying on venture capital is a dangerous model in that domain. I feel like the best options are:

1) Don't build a company around your FLOSS project, accept it'll be free in all senses forever

2) Start with a customer that will finance you and build slowly (focus on profit not revenue)

That's obviously not very helpful as finding that one customer is the really hard part so it's usually a hybrid of expecting what you build to be free forever and then finding that one customer. Quote/Charge a lot. Nope more than that. Now double the price and you're roughly there.

On the plus side you'll get very enthusiastic developers that tend to be intrinsically motivated which is a huge plus once you can build that company. I'd argue that this is one of the key competitive advantages of a "FLOSS company"

When it comes to new customers...Always be aware that you're not selling your cool tech but rather selling the solution to a specific problem (basically you're selling pain-easing medicine). I feel like it is beneficial to visualize a somewhat mean guy at the other end of the table who is only thinking this one line over and over "enough with the nerd speak, how does it help me and does it roughly cost what I expect it to cost". Yes we like to imagine that we can impress people with nifty tech details and shudder that the competition might be "that horrible Oracle database every developer hates". Hand in hand with this...charge a lot. The price serves as a signal. Just assume that evil guy you're talking to only cares about getting back to that golf course and won't even look at the number that closely as long as it's fitting his expectation. Yes that means charging too little can kill your deal.

I think the basic problem here is a lot of technology is heavily hyped and in use but 'not in use'. People are learning about them, trying them, even trying or using them in small internal deployments but are not buying them or supporting them in any way.

Those who take them into production are startups who are themselves trying to succeed or large companies who have tons of engineering talent who can create the product if need be but since its open source and available they use the open source version but they don't value it. Think the recent article where Github, forget supporting Redis has not even reached out to the developer once to thank him.

The demands of the community are high touch, even one or two weeks without commits and people start getting restive and raise 'alarms' on github and here. If there is potential value that you are still figuring out how to capture and a lot of attention you could end up expending time dealing with the politics of a fork. On the other end purchase decisions of most companies brings a whole new sets of requirements from internal processes, approvals, policies to decision makers and most open source companies do not have the resources or experience initially to manage the expensive and high touch sales and marketing process that most companies buying technology require.

If you have funding the pressure is even higher. There is a missing piece of how to translate support and usage of open source software for commerical reasons into some sort of revenue even minimal without friction. I think there should be a spotlight on companies using open source and their support and contribution back, and not just acquiring projects or hiring their developers which is an 'influence play'.

This was an excellent reading. No surprise given the author. I think there is a missing element about this in the post:

Correctness. We made very strict guarantees, and fulfilled them religiously.

It is true that the user base did not recognize this as important as the RethinkDB team, also because there is part of the development community which is very biased towards this aspect of correctness, at the point that you literally never experienced a problem for years with a system, but then a blog post is released where the system is shown to be not correct under certain assumptions, and you run away. So part of the tech community started to be very biased about the importance of correctness. But 99% of the developers actually running things were not.

Correctness is indeed important, and most developers have indeed less interest that they should have in formal matters that have practical effects. However, I don't think this point can be simply reduced to developers ignorance, that make them just wanting the DB to be fast in micro benchmarks. Another problem is use cases: despite what top 1% people say, many folks have use cases where to be absolutely correct all the time is not as important as raw speed, so the reason they often don't care is that they often can actually avoid caring at all about correctness, as long as things kinda work, while they absolutely need to care about raw speed in order to take things up with a small amount of hardware.