View all

In this blog post, we provide some details on how CockroachDB uses system memory on each node, and what you can do to keep memory usage in CockroachDB under control.

Some months ago I started work on a way to test random SQL statements with CockroachDB. This is important to expose unintended behavior in our server. For example, we want to prevent valid SQL statements from unexpectedly crashing a server or using all of its CPU or memory. We have already performed some small-scale fuzz testing, but fuzz testing often produces un-parseable input since it modifies bytes (although some fuzzers like AFL do attempt to produce clean input). The goal here was to produce valid SQL statements that the parser would accept and the system would then execute. These statements would essentially attempt to try various combinations of valid SQL to panic or otherwise render the system unusable (like consuming all CPU).

CockroachDB is a distributed SQL database built on top of a transactional key value store. We don’t (yet) expose the kv layer but it’s general purpose enough that we’ve used it to implement SQL without any special trickery. The particulars of how we represent data in a SQL table as well as the table metadata are internally called the “format version”. Our first format version was deliberately simple, causing some performance inefficiencies. We recently improved performance with a technique called column families, which pack multiple columns in one kv entry.

Scaling effortlessly over multiple nodes is one of the defining properties of CockroachDB. By maintaining a strongly-consistent database state across a network of machines, the distributed system can provide reliability and availability, while transparently tolerating disk, machine, and even datacenter failures. But not everyone has access to a datacenter at their fingertips, so we recently began looking into what it would take to run CockroachDB on a Raspberry Pi, one of the go-to tools of the modern day computer tinkerer.

CockroachDB is designed to be a scalable, survivable, and strongly consistent SQL database. Building a distributed system with these capabilities is a big task. Beyond the required functionality, it must also be correct, performant, and stable, or it isn’t worth the bits used to copy the binary.

Spencer Kimball

August 25, 2016

Until recently, CockroachDB’s SQL was suffering from a serious, long-standing bug – a schrödinbug, in fact – in its handling of table and column references. This blog post outlines how fuzz testing uncovered the error, how we discovered that our way of using Go was partly to blame, and how we addressed the issue using a form of strong typing.

As CockroachDB has grown over the last 30+ months of development, avenues for communicating between users and developers have proliferated. We started with GitHub, then created a Google Group, then an info@ email, then another Google Group, a Gitter room, and on we went. And while getting in touch with our developers is easier than ever (I suppose we don’t have Snapchat yet…), we hadn’t made a concerted effort to centralize our community’s brainpower – but that’s changing! Today we are launching the CockroachDB Community Forum, a place to ask how-tos, query others about best practices, and receive dev notices like our recent upgrade to Go 1.7.

Jessica Edwards

August 10, 2016

CockroachDB’s JOIN: An Early Implementation When our VP of engineering, Peter Mattis, made the decision in 2015 to support SQL, little did he know that the team would get as far as shipping the first implementation of CockroachDB’s JOIN exactly one year after that. A celebration is in order!

When you write data to CockroachDB (for example, if you insert a row into a table through the SQL client), we take care of replication for you. To do this, we use a consensus protocol – an algorithm which makes sure that your data is safely stored on multiple machines, and that those machines agree on the current state even if some of them are temporarily disconnected. In this post, I will give an overview of common implementation concerns and how we address these concerns in CockroachDB. Then I will abandon these earthly constraints and explore how we could improve consensus algorithms. Specifically, what would it take to make them faster?