Automatic Cloud-Scaling for Rapid Player Growth

Supporting global growth with flexible hosting

Rocket League is a unique “soccer with cars” game from video game developer Psyonix, part of the Epic Games family. Since its launch in 2015, the game has experienced steady growth in player numbers. 2020, however, was a record-breaking year. The huge boost in players can be partially attributed to the transition to free-to-play. With i3D.net’s newly launched Game Hosting Platform, Psyonix was able to host the influx of gamers without sacrificing latency.

How to host a latency-sensitive game exploding with player numbers

Rocket League is a distinctive game that has built a passionate community. “The aspects of the game sort of make it unique,” says Brian Jesse, server engineer for Rocket League. “It’s very easily understood by players. It relates to soccer. And the mechanics of the game are such that people don’t come into it automatically having skill, but you have to train your skills to get better at the game. It tends to grip people and then grow and gain in popularity.”

Rocket League’s growth is especially prominent in places like the Middle East, where it draws general sports and soccer fans more so than gamers. This spurs specific geographic hosting requirements for Psyonix. Especially when it comes to latency.

As a game played in real-time, high latency can have detrimental effects on the gamers’ experience. “Internet service providers are a little bit difficult to work with [in the Middle East], so it is important that we have a good connection” Jesse explains. “In some cases, other providers will route through Europe, rather than going direct to customers even in the same city, like in Dubai.”

Indirect trips for the data-packets, also called scenic routing, can contribute to latency issues. According to Jesse, Middle Eastern players have been vocal about getting better latency and server performance for some time. And it’s in the interest of game publishers to pay attention. In the end, having a better server experience is only going to improve the player population.

“i3D.net was really helpful, especially in the Middle East and South America.”

Brian Jesse, Server Engineer at Psyonix

Providing a low-latency gaming experience in all corners of the world

Amid a growing sea of cloud providers, i3D.net’s focus on high-performance, global bare metal infrastructure is important for latency-sensitive games like Rocket League.

In recent years, gaming infrastructure has moved more towards the cloud. Indie game publishers can jump in and have everything up and running. That does not necessarily work with a game like Rocket League. Cloud services tend to be centralized, rather than geographically spread out, serving a very broad region from afar. But for a latency-sensitive game, it is important for providers to be near the players.

i3D.net has provided hosting infrastructure for Rocket League since the launch of the game, hosting the game on multiple platforms including PC, PS4/5, Xbox, and Switch game servers. At the time of writing, i3D.net has over 40 Points of Presence around the world, with nearly a dozen more scheduled in 2021. i3D.net has made a point of opening in the developing regions often overlooked by major network providers. This brings the network as close as possible to the players. In the Middle East, for example, i3D.net has a presence in both Dubai and Fujairah.

Consequently, Psyonix was able to assure a stable, high-performance connection even in regions where connectivity is notoriously fractured. “So, I’m happy that you guys exist for that reason” laughs Jesse. “I think that i3D.net is the way to go in delivering on that.”

Leveraging i3D.net’s multi-cloud scaling platform for unexpected peaks

However, estimating player numbers and hosting requirements can still be a challenge. This proved true when Psyonix experienced record player numbers following their free-to-play migration.

“We were pretty good on what figures we thought we would see, so we definitely load-tested above the numbers that we got” Jesse explains. “However, certain regions like the Middle East and South America, especially São Paulo, all grew at a much more rapid pace than the rest of the world.”

Psyonix’s plan was to scale cloud instances to establish the baseline number, then fill metal instances to match. Several regions, however, blew up much faster than that, overshooting the original estimated peak point.

When it came to coping with the load it was important to get servers in those locations quickly. “I’d say that i3D.net was really gracious in giving the servers ahead of time, with contracts, moving as quick as i3D.net could. i3D.net was really helpful, especially in the Middle East and South America,” Jesse reveals.

Unexpected peaks or bigger growth than expected is when the cloud has its advantages. As Jesse explained, when places like São Paulo blew-up thanks to the free-to-play launch, instant bare metal is not always an option.

“Using the Game Hosting Platform, the time it took for Psyonix to patch all instances reduced from 1.5 hours to as little as 30 minutes with no impact on the gameplay."

For this new era in the Rocket League’s history, Psyonix needed a strategy where they could tweak the bare metal after bursting into the cloud for unexpected bumps. Their concerns surrounded the scalability of their backend services and keeping a stable, low-latency connection.

With the i3D.net Game Hosting Platform, Psyonix was able to create highly customized deployment environments on a static base of allocated bare metal servers to accommodate their everyday gaming traffic. And for those peak moments, they designated traffic to burst into multiple cloud accounts of their choice including Google Cloud Platform and Amazon Web Services.

Psyonix could set up cloud-burst locations of their own choice for each of their defined regions. Moreover, because they could designate multiple cloud choices for each location, the room to scale was always available even if one cloud provider was down.

Serving tournament player peaks throughout the day

The Game Hosting Platform also proved useful for Rocket League’s new Tournaments Mode. The mode enables players to compete against teams of a similar skill level, earning rewards in multiple competitive tournaments scheduled throughout the day. The demand on their servers was so high, Psyonix placed a player cap for each time slot.

“Tournaments get a very large player load very quickly. As soon as 3 PM hits, we’re filling those servers. Players are leaving the normal games right ahead of 3 PM and then all suddenly jumping in the tournaments,” explained Jesse. The result is an influx of push notifications from the backend, and a load on the database at the point where all the games are getting filled.

A planned feature for the platform is to anticipate the jump in capacity demand in tandem with the schedule. To handle the extra loads, the Game Hosting Platform spins up extra server capacity in response to players’ arrival. The scaling would then come into effect at the time of the tournament, for example at 3 pm, and then taper off until the next round, reducing instances between tournaments to cut costs.

Cutting server patching times in half

The Game Hosting Platform was also integral for server patching. “Free-to-play was a really complicated launch for us on the technical side” explains Jesse, “We had 3 hotfixes and then the actual free-to-play patch that all went out.”

Before the Game Hosting Platform, Psyonix had significant lag time because the system at their scale couldn’t keep up. The process of patching all game servers took around 1 – 1.5 hours.

Inversely, the Game Hosting Platform uses a highly parallelized, event-driven backend that can process game server deployment instructions in high succession. This is the equivalent of changing from using a single-lane road to a multi-lane highway.

Consequently, more patches can be done at the same time. Using the Game Hosting Platform, the time it took for Psyonix to patch all instances reduced from 1.5 hours to as little as 30 minutes. Most crucially of all, this was achieved with no impact on the gameplay.

Patching with no player impact

With the Game Hosting Platform, files are downloaded 15 minutes before the patch is set to start. Meaning a single instance would take under 5 seconds to patch.

However, this also includes a graceful shutdown and 30-minute timeout, enabling players to finish their game. For each instance, the automated patch waits for the players to leave before deploying the patch. The total timeframe for patching all 92,000 instances is spread over 30 minutes, with players able to rejoin a patched server within minutes.

“If we weren’t able to put those instances up quickly enough then we would run out of servers,” Jesse explains. Without automated patching, players would end up in a queue, clogging up the matchmaking and creating secondary effects on other systems.

Instant bare metal a click away

So, what does the future hold? Jesse is particularly excited about the latest i3D.net product offering; Flex Metal.

The Flex Metal service is a private bare metal cloud. It offers the flexibility of the cloud with the performance of bare metal servers. Flex Metal servers accommodate sudden increases in demand for computer power whilst keeping single-tenant occupancy. It’s appealing to Psyonix due to its hourly availability, providing flexible hosting with no trade-offs for efficiency.

“I am so excited for Flex Metal, it will bring so much value. I definitely want Flex Metal!” Jesse enthuses. “If we have instances where players are suddenly jumping on our platform, maybe there is a Fortnite event of some kind and players are coming over, or a big game shuts down by some sort of outage, we do see spikes in those instances. It would be so cool to offer that close-to-the-last-mile experience for those players that are suddenly jumping in quickly.”

Maybe the reason Rocket League keeps growing is simple – Psyonix continues to listen to and deliver on the needs of their community. From their move to free-to-play to their commitment to sourcing high-performance infrastructure in even remote regions, they’ve proved their dedication to giving the best gaming experience to everyone.

We had one final question for Brian Jesse. What do we need to do to improve our Rocket League Skills? “Play with people who are better than you. Play against a better opponent. And play lots and lots. Just hit the ball as much as you possibly can. It is free to play now. Download it.”

Ready to deliver seamless gaming experiences worldwide?

Get firsthand experience by requesting a free test server.

Company Description

Based in San Diego, California, Psyonix is a critically acclaimed video game developer behind some of the most creative and entertaining video games in the industry, including the hit sports-action hybrid, Rocket League.

Industry

Gaming

Compute

Cloud

Related Pages

Game Online Services

Related Pages

Connectivity

Related Pages

CDN

Related Pages

Colocation

Related Pages

By Application

Compute

Low Latency Connectivity

Content Delivery

DDoS Mitigation

Automatic Cloud-Scaling for Rapid Player Growth

Supporting global growth with flexible hosting

How to host a latency-sensitive game exploding with player numbers