Costly legacy challenges 

SumUp’s original infrastructure was built on AWS RDS for PostgreSQL. As the business grew, they started to face several challenges associated with scale, performance, availability, and maintenance. 

First, they were vertically scaling PostgreSQL to the point where there was nowhere else to go. Manual sharding is a well-known workaround, but causes a lot of operational overhead. Additionally, RDS for PostgreSQL has a single primary write architecture with one writer and many readers. This was becoming a performance bottleneck and was a single point of failure which put them at risk for outages.  

Not only were they concerned about unplanned downtown disruptions, but they were also regularly taking the database offline for routine maintenance, such as DB version upgrades. This type of planned downtime was no longer acceptable to SumUp, as they’re across multiple markets, continents and time zones. There’s no time for a scheduled downtime. 

Overall, operational complexity was becoming a problem at scale and they wanted a distributed system that would help solve some of these challenges out of the box. At a more granular level, these were their requirements for a new solution:

• Ability to horizontally scale out to multiple regions 

• Native change data capture (CDC) with first-class support for Debezium

• Online schema changes 

• Multi-node write support to scale traffic and maintain resilience

• ACID guarantees for correctness 

• Standard SQL support for developer efficiency 

• PCI-DSS compliant 

They started to evaluate several database options including CockroachDB, Yugabyte, ScyllaDB, and MongoDB. When they did a side-by-side feature comparison, CockroachDB stood out in a few particular areas: strict serializability, truly global multi-region writes, full online schema changes support, and native CDC. 

Looking under the hood

While the before and after diagrams may seem similar, there’s a lot of benefits that the SumUp team gained from CockroachDB. Since the migration, there’s been no data loss, and they’ve been performing zero downtime upgrades. They now have first-class support for Golang clients which was causing some operational challenges before. And, SumUp now has a self-healing architecture that automatically rebalances data to handle increases in workload. 

Today, they are utilizing CockroachDB for payment processing which is the main component of their core payment platform. They are also leveraging it for payments reporting, identity management, payments ledger, point of sale (POS), and other areas of the business that benefit from its capabilities.  

SumUp runs CockroachDB in a single AWS region, but they have plans to scale out to multiple regions in the future. Since the company operates in Europe, there’s several regulations such as the Digital Operational Resilience Act (DORA) that they are required to adhere to. Fortunately CockroachDB is cloud-native and will allow them to run their deployment across multiple clouds should the time come.

Embrace the cockroach way 

When switching to a new technology, there’s always some lessons to be learned along the way. Anton remarked that they needed to think differently about how a distributed database would work. He recommended embracing Cockroach Labs documentation, and to make sure you are following the best practices. 

When it comes to their new setup, Anton said it's smart to “treat CockroachDB as a first class citizen” and to integrate it into your observability stack. This will help you debug issues faster and always have insight into what is going on behind the scenes. He also recommends automating as much as possible to alleviate unnecessary operations for your team.