Just this past weekend, social and gaming platform Roblox saw a peak of 30.6 million concurrently active players, the company announced Tuesday. One game in particular—the record-breaking viral gardening sim Grow a Garden—drew a peak of 21.6 million concurrent players.
While previous blockbuster games from Fortnite to World of Warcraft primarily run on servers managed by their own developers and publishers, Roblox is distinctive in that its games and experiences are created by third-party developers. And those developers are free to update and tweak their game code at any time—with Roblox’s servers expected to manage the traffic load, even seamlessly updating the experience for players already logged in.
“Most of us computer scientists were taught that you’d never publish your entire code in one go, and you do it when your traffic is low,” says Anupam Singh, senior vice president of engineering at Roblox. “In our case, it’s almost the opposite.”
That’s because preannounced updates to big-name games naturally draw crowds of players, and no gamer wants to be stuck on an old version of the software in an era when screenshots rapidly circulate via group chat and social media. And since Roblox tries to avoid restricting how experienced creators run their games and when they can deploy them, game code, images, and other assets need to be sent quickly and simultaneously to Roblox’s content distribution network and edge servers as soon as they’re ready to go and certified to meet Roblox content standards.
It’s one of several challenges that have led Roblox’s engineering team to develop a sophisticated system of capacity and resilience planning, rigorous testing, and on-call engineering staffing for weekends, when players flock to the platform in droves.
The company has a network of 24 edge data centers around the world, handling much of the game experience. When players click a play button to launch a specific game, they’re connected to the most appropriate data center by an algorithm that can take into effect factors like which server their friends are playing on, their geographic location relative to the servers, and connection speed between the player’s device and each server. The system as a whole sometimes considers up to 4 billion combinations of players and servers per second, and the company has for years been optimizing the process with an ultimate goal of being able to handle 10 million players joining games in a period of just 10 seconds.
After all, today’s internet users are no longer used to loading delays in launching new content, especially not the younger users who make up many of Roblox’s core audience.
“We all remember the time when you just assumed that a little buffering is okay,” Singh says. “But there’s an entire generation of users who don’t think buffering happens on the internet.”
Those edge servers, plus additional cloud computing capacity that can be spun up to meet weekend demand, are connected to a pair of core data centers that manage services like the Roblox website, content filtering and recommendation algorithms, as well as the game publishing system. The edge servers connect to those core servers via a global private network, with redundant bandwidth available in case it’s necessary.
“I’ve learned in this job that cable being cut is a very regular occurrence,” Singh says.
During those busy weekends, there’s a rotating schedule of on-call engineers ready to respond to any incidents. Even C-suite executives participate, Singh says, with on-call workers expected to have a Roblox-approved computer and a good internet connection during those shifts. When the unexpected occurs, an incident manager leads the response, able to command everyone (including executives); infrastructure like AI transcription is in place for any necessary calls. The company strives to avoid casting blame to get incidents resolved properly and quickly, with incident managers empowered to approve resources as necessary to get the job done.
“The on-call has the ability to say, ‘Okay, give them 2,000 more servers, if that’s what’s needed right now,’” Singh says.
If a problem does pop up that limits capacity, the company has systems in place to gracefully scale services down, though it tries to avoid impacting players who are already engaged in a game, and won’t operate without some necessary features, like text content filtering.
On Monday, engineers with responsibility for code relating to any weekend incidents meet to discuss what happened, and on Tuesday, the company begins capacity planning for the weekend ahead. It’s also when Roblox observes TACO Tuesday, an acronym for “test actual capacity on Tuesday,” meaning engineers run tests constraining the resources available to code to ensure it runs properly under high traffic. Starting this year, Roblox has also rolled out a “chaos-testing” system, which deliberately injects errors, capacity constraints, and process restarts into the system to make sure it functions under stress.
Like Roblox game creators, engineers are also empowered to make updates to their code at any time, with hundreds of deployments possible during a weekday. And by Friday, the team is ready to roll out and test any needed extra cloud capacity based on demand projections for that weekend. Making weekly decisions about capacity is essential in a world where games can go viral in a short amount of time.
“Every three or four weeks, there’s a new big hit, so we’ve changed our capacity planning to be weekly,” Singh says. “And honestly, we would love for it to go to almost daily, where if there’s a hit within a day, we should still be able to find capacity.”
The extended deadline for Fast Company’s Next Big Things in Tech Awards is this Friday, June 27, at 11:59 p.m. PT. Apply today.
