Many customers approach the Cockroach Labs Account teams and the security and compliance teams with questions around vulnerabilities their preferred security scanners have identified. This blog post looks to demystify the space, explain what vulnerabilities are and more importantly, how Cockroach Labs triages vulnerabilities and addresses them in our daily operational cadence.
Vulnerabilities in the Wild
Image Source: https://www.bbc.com/news/technology-65814104. Accessed January 31, 2025.
Security breaches often feel inevitable, and for many companies that may be the case. However, taking a measured and objective approach to managing basic security risks can go a long way in reducing both the likelihood of, and the blast radius of, a breach. Patching vulnerabilities in the code base and staying on top of updates to dependent libraries are part of the fundamental building blocks of a robust security program.
Image Source: https://www.wsj.com/articles/hackers-exploit-log4j-flaw-at-belgian-defense-ministry-11640020439. Accessed January 31, 2025.
Cockroach Labs is committed to delivering the highest performing, most resilient transactional database platform on the market. One critical part of our software development process is a focus on Quality. The security of our code base is a key pillar of that process. Vulnerability Management is the practice of maintaining a critical and yet objective eye on possible vulnerabilities in the code base. This is table stakes for us as we secure our code. Without high-quality vulnerability management we would be comparable to a bank that leaves the keys to the vault on a hook outside the front door…Which to be clear, we are not!
What is a vulnerability?
Cribbing from ChatGPT: “a vulnerability is a flaw or weakness in a system (software, hardware, or process) that an attacker can exploit to gain unauthorized access or perform actions beyond what is intended or allowed.”
A real life analogy would be comparing a system or piece of software to a house. The house is the whole system in question and a vulnerability is any weakness that could allow someone to break-in or misuse the space. Anything that creates opportunity for intrusion, especially in a way that is not announced immediately to you, would be considered a vulnerability. Be it a broken lock, a window left open, or the lack of a well-established minefield on the front lawn.
The higher the likelihood that the weakness can or will allow access, the higher the severity of the vulnerability.
So, what is “vulnerable”?
To be clear, any system, software, or process likely has a vulnerability of some sort. The nuance is in the details. Take the house example. Imagine the house has a window left ajar that would easily allow an intruder inside. The house could be said to have a critical security vulnerability. However, if the window had a security screen on the inside that was not visible to the thief, the vulnerability from the open window would instead be low. Here, we can see that vulnerabilities may not always be as they seem from an outside perspective.
In relation to Cockroach Labs, and specifically CockroachDB, we have code that is developed in-house, which in turn may call on functions stored in external libraries (which are packaged into our binary). The binary also contains test code, which is code used to test the product but is never called by the “production” code. The base code and the dependent libraries (which are a multi-layered puzzle) are scanned nightly using tools we’ll discuss further below.
How are vulnerabilities tracked and rated?
Looking at CVEs (Common Vulnerabilities and Exposures) and CVSS (Common Vulnerability Scoring System) scoring are some of the first steps in understanding a vulnerability’s potential impact and urgency.
A CVE gives you a centralized and standard nomenclature and reference for a specific security issue. The CVSS takes that reference point for a vulnerability and assigns a numerical score (typically from 0.0 to 10.0 in version 3.x) that indicates how severe or critical that issue might be. This scoring can incorporate multiple factors:
The ease in which an attacker can exploit the vulnerability (exploitability)
Whether privileges are required
The perceived potential impact the vulnerability could have on a system (e.g., data loss, system takeover)
The maturity of the attack that could be used to exploit the vulnerability (Vulnerability Maturity)
All of these criteria are fed into the CVSS blender and a score is the key output. This can result in a “Critical” vulnerability with no known exploit path being considered less dangerous than a “High” vulnerability with a mature exploit.
Identifying exploitable vulnerabilities: SCA vs. SBOM scanners
Unpeeling that onion one layer further, scanners of various flavors are used to identify vulnerabilities in a software system. Two of the main ones used on non-running (or compiled) code are SCA (Software Composition Analysis) or SBOM (Software Bill of Materials) scanners. Scanners like these identify vulnerabilities in third-party libraries used by the software. The outputs of these scans can be extremely helpful in enumerating known issues, but they also raise the issue of reachability. A single library may include dozens or hundreds of classes, modules, or functions—yet only a fraction are actually called by the software. If the specific vulnerable function or code path isn’t invoked at all, that vulnerability is labeled Not Reachable, effectively neutralizing its threat.
This nuance is crucial: just because a library contains a vulnerability does not necessarily mean the application is exploitable by that vulnerability.
Finally, out-of-the-box scanners tend to flag every potential vulnerability within a library, often overstating the severity or listing vulnerabilities that cannot actually be reached. Without tuning these tools to match the application’s actual usage, you’ll inevitably see both false positives (vulnerabilities reported that cannot be exploited in your context) and inflated severity ratings. For these reasons any raw scan result must be taken with a proverbial “barge full of salt.”
It is best practice to combine automated results with manual triage, threat modeling, and (when needed) further testing of exploitability and reachability to confirm which findings are legitimate and which can be safely deprioritized or dismissed.
How does Cockroach Labs look at Vulnerability Management?
At Cockroach Labs, like at many companies, vulnerability management is a layered and complex space owned by multiple teams. The key areas of vulnerability management at Cockroach Labs include:
Vulnerabilities on endpoint devices (laptops, mobile devices, physical hardware, etc.)
Vulnerabilities in the Cloud stack that we own and manage as part of delivering the database-as-a-service (DBaaS) capabilities of CockroachDB Cloud
The vulnerabilities in the code that we write and own as part of CockroachDB
For the sake of clarity, in this blog, we are looking only at the last bullet: the management of vulnerabilities in the code that we deliver to customers under the brand of CockroachDB. Other areas will be covered in the future through more blog posts.
Preparing for launch!
CockroachDB is made up of in-house developed code, which in turn, leverages external libraries and functions. When it is released, that code is made available through multiple channels: direct access to the code on GitHub, the compiled binary on the company website, or through Docker images, where CockroachDB is delivered pre-installed on top of the latest Red Hat version, available through Docker Hub.
Cockroach Labs scans the GitHub code base on a nightly basis to identify new vulnerabilities and update the condition of any pre-existing vulnerabilities. Identified vulnerabilities are addressed as per our internal SLA for vulnerability management and the Security team is actively engaged with our engineering brethren to triage and remediate vulnerabilities that pop-up in the wild.
Our teams take a nuanced approach to accurately triage vulnerability reports from our SCA scanner and take a laser focus on only the vulnerabilities that could actually be exploited, reducing the toil and disruption to the engineering development process. We triage the vulnerabilities based on their CVE classification and the relative CVSS scoring. The initial indication of severity is then weighed against the code base itself:
The answers to these questions, and others, are weighed against the base severity, allowing the security team to reclassify vulnerabilities as being False Positives or being of much lower severity than baselined. Bringing us back to the home security analogy, you could have a window without a lock, or one that is left open, but if it’s on the third floor, accessible only by ladder, then the likelihood of a breach is low to none. The details always matter. .
What about Docker images?
When a CockroachDB version is ready for prime-time, we deploy it as previously described. For Docker images we have that next layer: OS image vulnerabilities. At build time of the Docker image, we download the latest Red Hat image, apply all applicable and available OS patches, install the CockroachDB binary onto the image, and then publish it.
OS vulnerabilities can, and do, appear after the publishing of the image, but Cockroach Labs is unable to continuously update and release versions of the Docker image based on those changes in the wild. For this area of vulnerability management, we assume customers will treat the Docker image as any other asset within their environment, and will apply company-level patch management processes to that asset. OS image vulnerabilities will be patched upon the release of the next minor, or major release of CockroachDB, but not in-between.
So…what now? Continuous review and improvement
Vulnerabilities are always there. Security scanners are effective at finding anything that could cause a security breach but they lack the delicacy that internal security organizations bring to understanding the target system. An identified vulnerability does not mean the system is inherently exposed, just that it could be. Cockroach Labs takes extremely seriously all concerns related to security, including vulnerability management. We will continue to work towards reducing and eliminating any and all vulnerabilities that could put our customers and our employees at risk of loss of any sort. As with any security control, we are committed to continuously reviewing and improving our processes and strive to deliver the most robust, resilient, and secure code to both our self-hosted and to our cloud customers.
Learn how you can enable modern, resilient applications with CockroachDB: speak with an expert today.
