CockroachDB v23.2 now seamlessly integrates with Confluent, using change data capture so users can connect the CRDB's distributed SQL capabilities with the real-time event streaming features of Confluent.

Exporting data is a crucial tool in any database user’s toolkit. In CockroachDB, the EXPORT command has long provided this essential functionality to: Move data to a different data store for business analytics Migrate data to a new database Archive data in a platform-neutral way Seed an application with data Say, for example, you need to export a sizable amount of JSON to seed a core data store for a streaming service. Sometimes you can pre-seed this data alongside your application with an EXPORT. But if you’re working on the scale of a company like Netflix, EXPORT commands come with some limitations that led us to explore an additional way to export data.

Exporting data is a crucial tool in any database user’s toolkit. In CockroachDB, the EXPORT command has long provided this essential functionality to: - Move data to a different data store for business analytics - Migrate data to a new database - Archive data in a platform-neutral way - Seed an application with data Say, for example, you need to export a sizable amount of JSON to seed a core data store for a streaming service. Sometimes you can pre-seed this data alongside your application with an EXPORT. But if you’re working on the scale of a company like Netflix, EXPORT commands come with some limitations that led us to explore an additional way to export data.

CDC Queries are SQL-like statements that allow you to (1) filter (2) transform (3) and choose the schema of your data stream. Instead of the headaches of sidecar services or downstream tools, filtering and transforming streaming data is now as simple as a SQL query. Here is a quick video tutorial about how to use CDC Queries that accounts for all the new functionality as of the CockroachDB 23.1 release:

Moving and processing data between systems is a common pain point. Users need up-to-date data across systems for use in business analytics, for event-driven architectures, for creating audit trails, or for archiving data. One way to do that is to set up an external service that regularly polls the database for changes.

Whether you are streaming to an analytics platform for business intelligence or building event-driven services, CockroachDB’s change data capture (CDC) capabilities are powerful and adaptable to your application needs. But how can you leverage your changefeed setup to get the performance that best fits your application? Most changefeed settings & cluster settings outlined here will have some tradeoff associated with them. Going in, you should have an understanding of what you are targeting and compromises you are willing to make. We recommend testing your configuration under your workloads!

Batch data sucks — it’s slow, manual, cumbersome, and often stale. We know because we’ve dealt with these problems ourselves. Any business needs to track metrics, from customer activity to the internal workings of the company. But how can we keep those metrics up to date to extract maximum business value? Here at Cockroach Labs we build CockroachDB, a distributed database meant to survive everything and thrive everywhere. CockroachDB is primarily optimized for transactional- “OLTP”- data, and sometimes it is advantageous to stream that data to an analytical warehouse to run frequent, large queries. That’s what we do with our “telemetry data” — the data we collect internally on product usage.