View all

Almost a year ago, we wrote about our use of Jepsen in testing CockroachDB. As we prepare for CockroachDB 1.0, we wanted to get independent verification of our findings, so last fall we hired Kyle Kingsbury, the author of Jepsen, to review our tests and add more of his own. Last week, Kyle published his results. Kyle’s testing found two new bugs: one in CockroachDB’s timestamp cache which could allow inconsistencies whenever two transactions are assigned the same timestamp, and one in which certain transactions could be applied twice due to internal retries. Both of these issues were fixed (the first in September and the second in October), and the expanded Jepsen test suite is now a part of our nightly test runs. Now we can say with more confidence that CockroachDB meets its guarantee of serializability (the highest ANSI SQL isolation level) for all SQL transactions.

Six months ago, we reported our first implementation of SQL joins in CockroachDB. At that point in development, we merely provided a functional proof of concept that was severely limited performance-wise. This is changing, gradually. We are preparing to launch CockroachDB 1.0 later this spring, and SQL joins in CockroachDB 1.0 will be usable, if underpowered.

One would think that designing for creative consumers through a massively popular brand would be a dream come true for any young designer. After all, what more could you want than a great company, compelling product, and an engaged team? A bigger challenge, it turns out.

Building a complex system requires writing and debugging many tests. Developer tests that can be run without any infrastructure or configuration are crucial for allowing fast-paced development while avoiding regressions. Even for a distributed system like CockroachDB, tests that run on a single anode (e.g. as part of go test) can cover many aspects of the system: one technique we use is that of creating a virtual test cluster with multiple CockroachDB nodes, all running within the same process.

CockroachDB is a project I’ve been keeping an eye on for a while. It’s a an open-source, Apache 2 licensed, database system(Github link). At it’s core its a key-value store that scales horizontally. But what makes it really interesting for us though, is that 1) it supports SQL by using the Postgres wire protocol and 2) has full ACID semantics and distributed transactions. If you’re interested in how they achieve this, make sure to read the technical posts at the CockroachLabs blog (I admit, sometimes it’s not for the faint-of-heart ;-)). Do note that it is still a distributed system and thus follows the CAP theorem, more specifically it is a CP system.

If you’re a fan of DigitalOcean and its powerful but simple platform to deploy cloud-based infrastructure, you’ll appreciate CockroachDB: it’s similarly simple to deploy and provides your stack a lot of power and flexibility. And while CockroachDB can be deployed anywhere, it’s a natural fit within DigitalOcean’s no-fuss framework: both are for developers who like easy-to-reason-about technology that lets them get work done quickly.

In August, we published a blog post entitled “Why Can’t I Run a 100-Node CockroachDB Cluster?”. The post outlined difficulties we encountered stabilizing CockroachDB. CockroachDB stability (or the lack of) had become significant enough that we designated it a “code yellow” issue, a concept borrowed from Google that means a problem is so pressing that it merits promotion to a primary concern of the company. For us, the code yellow was more than warranted; a database program isn’t worth the bytes to store its binary if it lacks stability. In this post, I’ll set the stage with some background, then cover hypotheses for root causes of instability, our communication strategy, some interesting technical details, outcomes for stabilization efforts, and conclusions. It’s a long post, so bear with me!

Spencer Kimball

November 16, 2016

In this blog post, we provide some details on how CockroachDB uses system memory on each node, and what you can do to keep memory usage in CockroachDB under control.

Some months ago I started work on a way to test random SQL statements with CockroachDB. This is important to expose unintended behavior in our server. For example, we want to prevent valid SQL statements from unexpectedly crashing a server or using all of its CPU or memory. We have already performed some small-scale fuzz testing, but fuzz testing often produces un-parseable input since it modifies bytes (although some fuzzers like AFL do attempt to produce clean input). The goal here was to produce valid SQL statements that the parser would accept and the system would then execute. These statements would essentially attempt to try various combinations of valid SQL to panic or otherwise render the system unusable (like consuming all CPU).