When Friday the 13th, a video game adaptation of the slasher film franchise, first launched, gamers booted it up hoping for the same kind of horror experience they’d seen on the silver screen.
But instead of Jason Voorhees’s creepy mask, many of them were greeted by a different sort of scare: Game Database Login Failure.
The cause, according to the developers? A database that just wasn’t ready:
We had upwards of 75K players in a short period of time. Due to this our database was hit hard. We tried spinning up more nodes to help alleviate the stress but ultimately caused some corrupted nodes which is causing people not to be able to login that were affected by those nodes. In addition it has caused major slow downs in our database software which can on and off effect other accounts from logging in.
Databases may not be the most exciting part of game development. But as Friday the 13th and many other games have demonstrated, they’re a critical part of the infrastructure for any game with online services.
So what do game developers need from their databases? We’ve learned from developers like Devsisters and Mythical Games, as well as gaming platform providers like Insights Gaming and Heroic Labs who are using CockroachDB to power games with hundreds of thousands of concurrent users and tens of thousands of transactions per second.
Here are some of the database features that are most critical for game developers:
Automated scaling
Games can generate some pretty spiky swings in workload. And as the Friday the 13th example aptly demonstrates, trying to predict demand ahead of time is a tricky proposition. Getting it wrong has a hugely negative impact on player experience.
That’s why it’s critical to choose a database with automatic elastic scaling. This allows you to scale up instantly when demand spikes. The player experience isn’t affected, and the developers don’t have to run around frantically trying to spin up nodes to handle the load whenever the player count climbs. Having automated scaling can also help keep costs down by spinning down extraneous nodes when they’re not needed.
Devsisters, for example, uses CockroachDB to power their game Cookie Run: Kingdom in part because they knew it would need to handle spiky workloads. When they release new content, player traffic doubles, and transactions can spike as high as 60,000 per second.
“Looking back, it would have been impossible to scale this game on MySQL or Aurora,” says DevSisters’s Sungyoon Jeong, who’s a part of their DevOps team. “We experienced more than 6x the workload size we anticipated, and CockroachDB was able to scale with us throughout this journey.”
Low latency
Nobody likes waiting around on a database, but gamers are a demographic that are particularly conscious of latency. That can present a challenge because they’re also typically distributed all over the globe. While a retail payments company can focus on launching in a specific geographic region, exciting games will generate interest all over the world. Gamers on every continent will expect a smooth, low-latency experience.
At a minimum, that means that your database must be highly performant. But often, it also means that it must be multi-region. If a player in Australia has to ping a database node in US East for every single transaction, state change, etc. their experience is probably not going to be pleasant.
Getting this right means setting up a performant, multi-region database configuration. Devsisters, for example, uses a follow-the-workload configuration for CockroachDB: CockroachDB replicates Devsisters’s data across multiple regions, but the “leaseholder” — the data replica that coordinates reads and writes for each range of data — moves automatically to follow real-time demand, so that when player numbers in Korea spike (for example), they’re reading and writing to a database that’s nearby, not halfway around the world.
Low latency is also one of the reasons Insights Gaming, a collaborative esports coaching platform, chose CockroachDB. With an audience that includes professional players and coaches, Insights knew that any lag on their platform would frustrate their users. That’s why they use CockroachDB for all of their latency-sensitive data, including account info and access, coaching comments and annotations, payments, etc.
Consistency
Games can generate heavy transactional workloads. And since many modern games are funded partially or entirely by in-game transactions like item purchases, there is little room for error here — game developers must often track transactions with the same level of accuracy as financial institutions. Both revenue and player experience can be significantly impacted if transactions aren’t consistent and accurate.
It’s not all about transactional workloads, either. Many games have services workloads that store and serve metadata about players for services like leaderboards, player profiles, player statistics, etc. Consistency and correctness are essential for these workloads as well – gamers are competitive, so any kind of performance-tracking has to be both available and accurate.
This presents a challenge, because some of the other database needs we’ve discussed – elastic scaling and multi-region capability – don’t always play nicely with consistency. In particular, NoSQL databases offer an excellent solution for global scalability, but are unable to offer ACID transaction guarantees, which makes them a poor choice for any game with important transactional workloads or other consistency needs. On the other hand, traditional SQL databases offer ironclad ACID transactional consistency, but are difficult and operationally intensive to scale.
Heroic Labs, a provider of the game infrastructure that powers the games of developers Zynga and Paradox (among many others), chose CockroachDB in part because it offers a “best of both worlds” solution to this problem.
Heroic Labs’s Nakama servers include digital wallet APIs that make it easier for game developers to offer in-game purchases, so the lack of consistency guarantees offered by NoSQL databases was a dealbreaker. But at the same time, they also needed automated elastic scaling and fault tolerance, which ruled out using a traditional relational database, too.
The solution was a distributed SQL database: CockroachDB gave Heroic Labs a database that was easy to deploy, easy to scale, and highly consistent, which made it the right choice to package with their Nakama server product.
High availability
Finally, game developers often need a database that can provide high availability. This isn’t particularly unique — database downtime is not considered acceptable in any industry these days — but particularly for the developers of free-to-play games, database availability can be a matter of life and death.
Consider, for example, the story of Friday the 13th from the beginning of this article. Players had already spent money to purchase that game, so although they had a negative experience the first time they tried to play and some of them probably asked for a refund, many of them likely returned to try it again later.
Free-to-play games don’t have that “sunk cost” factor to bring users back if they have a bad first experience, though. If a game doesn’t work smoothly the first time, chances are that a player is not going to interact with it again — except perhaps to leave an angry one-star review!
A database outage during a high-traffic moment like a launch could spell doom for a free-to-play game, undoing months or years of a developer’s hard work.
Resiliency is one of the other reasons Heroic Labs chose CockroachDB as the database to power its Nakama game server product. No customer is ever going to be satisfied with unscheduled database downtime, and many of the developers that use Nakama servers are focused on free-to-play games where database availability is almost as important as transactional consistency.
Takeaways
Although choosing a database technology might not be the most exciting part of game development, database infrastructure can have a huge impact on a game’s success, so it’s important to choose wisely.
While no two games will have the same exact requirements, if you’re working on a game with a potentially large audience and any kind of online service, it is likely that you will need a database that can:
Scale automatically without adding operational complexity
Serve players across the globe with low latency
Guarantee ACID transactions for bulletproof consistency
Survive node, AZ, or even regional outages and provide five-nines availability
CockroachDB ticks all of those boxes, which is why it’s trusted to power services in truly massive online games like Cookie Run: Kingdom. If you’re developing a game, why not give it a try?
