One of the hardest-rocking presentations at RoachFest 2023 came from the supergroup of Joe Rizzo and James Lupolt, application architecture frontmen for Hard Rock Digital. The two described how Hard Rock Digital, the online sportsbook and iCasino, came to choose CockroachDB as their data platform. Eventually, that choice enabled Hard Rock to reduce total cost of ownership for their betting app by avoiding duplicative infrastructure while complying with a complex web of gaming industry regulations. This post covers the main topics from Rizzo and Lupolt’s presentation. For detailed technical descriptions and information, check out the recorded presentation and slide deck.
The Wire Act
“In online real-money gambling, there’s a lot of regulatory requirements that dictate your application architecture,” explained Lupolt, a DBA for Hard Rock Digital. “The federal Wire Act prohibits players from wagering across state lines, and then each state has its own regulations.”
The challenges they faced included:
Specific transactions must take place in a geographic location. For example, placing a bet must transact only in that particular state or territory, if governed by a particular tribe.
Certain data can only be in certain locations
Certain data can be only updated from certain locations
Initially, to safely satisfy the plethora of regulatory requirements, the team considered creating and maintaining a separate database in each state where they operated — but the TCO of eventually doing this in all 50 states was astronomical. So how did Hard Rock solve this compliance problem and have a single unified data platform that can be available in multiple states? They weren’t sure, exactly…That is, until they found CockroachDB.
Many locations, one database
“Instead of many databases spread across the United States, we have one logical database, one logical system, one thing that we’re managing and running regardless of the jurisdiction,” said Senior Platform Architect Joe Rizzo. “Instead of investing all the capital to manage and monitor and run these disparate database installations, we can just collapse everything into a single CockroachDB cluster. And then distribute the nodes to the geographic locations where they’re needed to meet the residency requirements, the data residency and compute requirements.”
Depending on a given state’s regulations, he explained, this means putting a CockroachDB node in an AWS region on EC2. In locations with stricter limits on data location, however, sometimes there is no AWS region local enough to where betting transactions must take place. In these cases, the team uses Amazon outposts as a way to extend AWS into their own data centers. “We install Cockroach on EC2 running in the AWS outpost,” said Lupolt, the Hard Rock Digital DBA. “That solves some significant problems for us while reducing infrastructure costs a lot.”
CockroachDB is at the core. That’s the life, that’s the blood, that’s the crown jewels. Our application runs on top of Cockroach, and then underneath the compute runs on AWS Cloud Continuum.
Lessons learned
Rizzo and Lupolt took turns sharing the learning they’ve gained over the past several years of launching and running online sportsbooks (thus far in seven states, and more to come) with CockroachDB.
“While we were building this, we learned a number of lessons,” said Lupolt. “In a lot of cases, the way we thought initially was the best way to do things ended up not being optimal.”
Here is a quick overview of lessons learned. Watch the presentation video for technical guidance (with diagrams!) on solving these issues.
Lesson one: Beware of unnecessary decentralization
It may sound obvious but it took us a while for us to get our heads around the fact that, just because you’re using a distributed database doesn’t mean you have to decentralize everything. For example, we eventually realized that a good solution for certain data types was to find a location that’s in the middle of where all your customers are. Then you have roughly the same average latency to everyone, and then put some CockroachDB nodes there and do data processing there. (James Lupolt)
Lesson two: Keep a close eye on network changes
The network is the backbone of an interstate distributed system. But when we’re deploying outposts plus on premise plus at various locations like casinos — even though it’s 2023, the networking is not always the best.
We learned, after going through this a lot of times, that we need to monitor changes in the networking. Not just is it up or down, but is there flapping, is there packet loss, is there latency? Also worth noting is Cockroach nodes need the ability to connect to every other Cockroach node. Not that that always happens, but when you think about monitoring, understanding your replication strategy is key in building your network monitoring. (Joe Rizzo)
Lesson three: Query your query performance
Another thing we learned: In a more traditional database, when you’re analyzing query performance, you primarily think about things like the shape of the query plan, your schema and indexing strategy and how fast your disks are. All of these still matter in a distributed SQL database, but now the network becomes much more prominent. You definitely have to keep the network in mind more when you’re analyzing query performance and analyze how many round trips you’re making between the database client and the gateway node or between the gateway node and the lease holder. (James Lupolt)
One database to rule them all
In closing, the speakers reiterated how CockroachDB gave Hard Rock Digital exactly what they needed: a single unified data platform that could potentially be available anywhere but also satisfy complex regulations.
“The traditional way of deploying sportsbook apps is with a completely independent silo in each jurisdiction," said Rizzo. “With CockroachDB, because we don’t have to duplicate services, we have faster time to market with lower costs — and we give our players a better experience.”
