Data loss shouldn’t keep you up at night: Rest easy with always-on security at enterprise scale

We live in a world where companies are built around helping other organizations avoid risk. And one of the best ways to do this is by helping them safeguard their most important asset: data.

The majority of the time, data loss happens by accident (primarily human error and system failures) not because of malicious intent or bad actors. An employee neglects to update their software, an entire cloud region goes down, there’s an unexpected influx of traffic that the company wasn’t prepared for – we’ve all heard or have been part of these stories in some way or another.

Over the years, data protection has become even more complicated because businesses are running so many applications in the cloud that employees are accessing on a daily basis. The more applications you have, the greater the risk.

In 2018, Nightfall was founded to help organizations prevent data loss and protect data across their cloud applications. It uses machine learning techniques to detect and classify customers’ sensitive data. The Nightfall platform allows you to inspect data anywhere it lives (in the product, data flows, or third-party apps) and eliminates the need for developers to build their own sensitive data detection logic from scratch.

For example, if a company is using Slack, Nightfall offers a Slack integration that can identify sensitive data such as social security numbers or credit card numbers, then send alerts to the appropriate people letting them know that sensitive information is at risk. It’s especially useful for companies that have to adhere to compliance regulations around customer data hygiene, such as HIPAA.

There are two main components to Nightfall’s product offering: 1) integrations with popular cloud applications like GDrive, Slack, GitHub, Jira, Confluence and 2) a cloud-agnostic platform that uses AI/machine learning to identify sensitive data. The Nightfall Developer Platform is a set of APIs that allow you to build data protection features into any app and protect customer data across its lifecycle, starting at ingestion.

Many of their customers are generating terabytes of data, and those scan request volumes can fluctuate suddenly depending on their business operations (customers have the ability to upload any amount of data at any time and request scans). Given this need for tremendous scale while still ensuring an ideal customer experience, they turned to CockroachDB as a backend for their API gateway and interactive analytics dashboard.

When Nightfall started building their product 4 years ago, they wanted to deliver an MVP quickly and they wanted it to be cloud-native. At the time, a majority of the tech they were using was chosen by the company’s co-founders – tech that had worked well for them in the past.

They were running a couple different databases for different use cases. They were using a bit of Cassandra, Postgres, RocksDB, and TimescaleDB.

And this all worked well at first until their customer base started to grow. Not only was managing several different databases a headache and inefficient use of time, but they needed a solution that could scale horizontally to accommodate more customers. They also wanted to consolidate their databases down to one and have a single source of truth for all of their data.

Around the same time, they started hiring more engineers to help support business growth now that they had found product-market-fit. They decided to migrate from Docker on EC2 to Kubernetes. Given the distributed nature of Kubernetes, they now had the ability to operate applications at scale.

To take full advantage of that, they needed a database system that could match the distributed scale Kubernetes could provide. They wanted a solution that…

• Delivered high availability so that their customers’ data was always available

• Scaled out to multiple regions for so their customers would have low latency access and they could meet global data regulations

• Allowed for easy horizontal scale to keep up with their growing user base and could support unpredictable spikes in traffic

• Lowered the learning curve for developers and provided SQL compatibility

They started to shop around for distributed databases, keeping in mind that they wanted a cloud-native, agnostic solution. After coming across CockroachDB, they were impressed with its ability to scale across multiple regions, which the team thought would help them unlock new locations in the future while still meeting data locality requirements.

“We are looking to expand into EMEA in the near future and are aware of the complications that can arise when entering new territories. CockroachDB provides tremendous value when it comes to scaling your app across multiple geographies. It handles the complexity in the database layer, so we wouldn’t have to deal with it in the application layer.” - Dan Hertz Platform Team Manager

After evaluating CockroachDB, they realized how easy it was to spin up a cluster and when they are ready, span across multiple regions. Since they are in the security space, they are handling a lot of sensitive data while trying to mitigate regulations. Having the ability to control where data resides is a huge benefit.

And CockroachDB fit seamlessly into their software delivery supply chain model. They use a variety of AWS tools like Lambda, Glue, and SQS alongside their Kubernetes clusters. They dogfood their own developer platform with CircleCI testing to scan their own code for sensitive data like PII or API keys. They push this data to their container image registry and then automate their deploys with Terraform.

They use Kafka for streaming data and Kafka queues to help manage large influxes in traffic. They also use Temporal (for workflow management) on top of Kafka that allows them to more predictably conduct asynchronous operations so all their jobs get completed on time.

They are running 5 CockroachDB clusters in a single AWS region now which is in close proximity to their current customer base which is primarily in the US.

It won’t be long before Nightfall transitions into a multi-region setup so that they can accommodate a user base in EMEA. Then, their deployment will look something like this:

As you can see from the diagram, they scaled their application to two additional regions, but CockroachDB still functions as a single logical database.

The Nightfall team is running two major use cases in production on CockroachDB. The first is using CockroachDB as a metadata storage layer for their API gateway (which they built themselves). They have a high volume of API requests coming into their system and they need to return quick responses to confirm questions such as… is the API key valid? Is this customer within their quota for the request they are trying to make?

They use CockroachDB because it can deliver performant reads AND writes. For example, they might have 100 requests coming in per second and they need to write request logs for each of them and read out quickly to see if the API key is valid. They also use CockroachDB to store key entities such as telemetry, usage tiers, rate limits, request logs, etc.

“We have very high bandwidth use cases that require the ability to deliver performant reads and writes. With CockroachDB, we’ve never had to worry about any bottlenecks and have been impressed with its ability to scale as our customers generate tremendous amounts of data.” - Evan Fuller, Engineer

Their second use case is an interactive analytics dashboard that uses CockroachDB as the system of record because it can support a high volume of data ingestion. The data generated from Nightfall’s integrations with Google Drive, Slack, and other cloud apps can spread out among many different locations. Since CockroachDB functions as a single logical database regardless, they have a centralized place to host information even when they scale out their application to multiple regions. And because CockroachDB delivers high availability, their customers can always view/interact with their data.

Like the API gateway, this is not just a read-only use case because of the interactivity element. Users can take action on what they see in the dashboard and respond to alerts. They pull analytics straight from CockroachDB and use change data capture (CDC) to write to S3. CDC ensures that they are providing the correct data for their customers.

The Nightfall team said they felt very comfortable with PostgreSQL, which was actually a huge selling point for CockroachDB. Their team could reuse their SQL knowledge and get started quickly.

The API use case was pretty straightforward given CockroachDB’s high read and write bandwidth. However, their analytics dashboard was a little more complicated to build. It’s important to remember that CockroachDB is a distributed database and has unique capabilities that traditional relational databases do not have.

The Nightfall team has a few tips they recommend you keep in mind:

• If you are performing complex queries with a lot of JOINs, you can end up with hotspots and subpar performance. You can make better index choices to improve SLAs.

• If you need to control where data is distributed to, make sure you set up primary keys properly and understand how follower reads work.

• Remember that CockroachDB has ACID guarantees which can have an impact on code design.

Nightfall worked with the Cockroach Labs architects to help them develop a setup that worked best for their analytics use case. If you are migrating from PostgreSQL to CockroachDB, here’s some other tips to keep in mind.

Nightfall’s engineering team believes that you should “pick the right tool for the job, but also, don’t be scared to try something new.”

Since they’ve migrated to CockroachDB they’ve been able to “focus on building their product and spend less time on unnecessary operational tasks” that come with managing other databases. They plan to expand their presence to EMEA within the next calendar year and will use CockroachDB to support this goal.

If you are interested in trying out Nightfall’s platform for yourself, you can sign up for free. And if you like to build with modern database tech like CockroachDB, they are hiring.