Writing

One platform, 100 million players: what game launches taught me

Game launches are the purest stress test in software. The date is fixed months ahead. Marketing has already told millions of people. And when the servers open, everyone turns up within the same few minutes. Fans don't get a soft launch, and you don't get a second first day.

From 2017 to 2021 I built and led the game studio services group at The Pokémon Company International in Bellevue, Washington. What we started with was a working prototype, a monolith that served one game well. By the time I left it had become a multi-tenant, multi-region platform serving more than 100 million monthly active players across Pokémon's games, on mobile, TV and the web, with consoles in the design from the start. The whole platform was built to a five-nines availability target.

One of the games we built it for is Pokémon Trading Card Game Live. It replaced Pokémon TCG Online, which had been running since 2011. The company announced it in September 2021, shortly after I moved on, and launched it worldwide in June 2023 on iOS, iPadOS, Android, Windows and macOS, with cross-platform play. My team built the services underneath it: accounts, collections, matchmaking and purchases. So when someone starts a game on their phone and later picks up the same collection on a laptop, that's our work doing its job.

Not every customer was a game. The Pokémon TV app, where fans watch the anime, had originally been built by a third party. Our studio brought it in house and rewrote it completely, app and all, and we built the game studio services from scratch to run it and the titles that would follow. We took the new backend from a code-complete alpha in August 2018 to launch that October. Two months, start to finish.

The third tenant is one most players never see. Organizers use the official tournament app, on mobile and the web, to set up and run the Pokémon World Championships. It ran on the same platform, and it made its debut at Worlds 2019 in Washington, D.C.

Michael Hrádek with his all-access badge for the Pokémon World Championships 2019, a giant inflatable Pikachu behind him All access at the Pokémon World Championships, Washington, D.C., August 2019, where our tournament app made its debut.

Here's what those four years taught me about building something other teams bet their launches on.

1. The second game is the real test

The prototype did what it was built to do. It let one team ship one game. About a year in, my charter grew to cover any number of titles and products, and that's when one game's worth of design stopped being enough. Title number two wants the same login, purchases and leaderboards, but with its own data, its own release dates and a traffic pattern nothing like the first. Copy the monolith and you now have two monoliths to keep alive.

So the first big decision was to make multi-tenancy part of the platform itself. Every service knew which title, version and region a request belonged to, and data was partitioned by tenant from day one. Adding a product became a matter of configuration and an onboarding checklist. That's how TCG Live, Pokémon TV and the tournament app ended up sharing one platform.

A whiteboard from those years shows what "platform" meant in practice. Next to the obvious pieces is everything a live game needs and no game team wants to build for itself: barcodes, friends, messaging and an inbox, saved game state, sales and promotions, an expiration service, multi-region admin. And one line that turned out to be far more important than it looks: import TCGO, so players could bring their Pokémon TCG Online collections with them.

A whiteboard listing platform features: barcodes, friends, messaging and inbox, admin as product, import TCGO, blue/green, push to prod, sales and promos, expiration service, save game state, multi-region admin Planning the platform: the features every game would share.

Deciding what version 1 would leave out was just as important. We mapped it on sticky notes, one per service. The store handled the codes printed in physical card products, plus in-app purchases and booster packs. Inventory granted items and checked that a deck only used cards the player actually owned. Battle covered WebSockets, match logic and matchmaking. Then came profile, and admin. Two of the notes carry the word that made the date possible: cut. Messaging and friends moved to a later version.

Sticky notes mapping the first release: store, code redemption, battle, inventory, admin, profile, and messaging and friends marked as cut Scoping version 1, one sticky note per service.

Two of those services show how a platform earns trust. Inventory was a ledger. Every grant, purchase and redemption went in as an entry you could audit and replay, so any count in a player's collection could be explained. The collections API lived in the client SDK, but whether a deck was legal got decided on the game server, through the server SDK, against that ledger. Someone can always tamper with a client. They can't change the server's answer. Matchmaking followed the same rule and paired players on deck strength reported by the game server, ignoring whatever the client claimed.

One phrase on that whiteboard became a principle I still use: admin as a product. Live games run on their tools. Promotions, sales, events, player support: all of it happens in admin consoles. Leave those as an afterthought and every live-ops change turns into an engineering ticket. We gave them real users, a roadmap and the same quality bar as the services, and that's what let the game teams run their own games without waiting on us.

2. Design for launch day

Average traffic is a comforting number, and it tells you almost nothing about launch day. We designed every service for the spike:

  • Absorb bursts. Queues (SNS, SQS and RabbitMQ) sat between the edge and the services doing the real work, so a surge turned into a backlog. You can drain a backlog. An outage you have to explain.
  • Keep the hot path cheap. Player state lived in DynamoDB, with Redis and Memcached in front of anything read far more often than it was written.
  • Contain failure by region. With multi-region deployment, a regional problem hurt one part of the world and left everyone else playing.
  • Degrade on purpose. Decide ahead of time which features can switch off under load. A leaderboard can wait; logging in can't. Build that as a switch so nobody has to make the call at 3 a.m.
  • Plan for messages going wrong. Real-time play over WebSockets meant designing for dead letters, stale queues and throttling between services from the start. They had their own column on the planning whiteboard.

Being ready for launch also came down to how code reached production. Before building the pipeline, we wrote its requirements on a whiteboard. A few of them shaped everything else. Every merge to master was tested and deployed to dev automatically. Getting into the environments that counted took a push of a button, behind a gate. And we always had to know what was running where: which services, SDK versions, game engines, tables and dependencies, and what had been deployed when. Each build produced versioned artifacts (service containers, plus client and server SDK packages), and that same artifact moved from one environment to the next. Nothing was rebuilt along the way.

A whiteboard listing deployment requirements, from every merge to master being deployed to dev, to blue/green, draining and ordering Requirements for the deployment pipeline, written before any of it existed.

Then there was the question of where code ran before production. Each audience got its own environment with one job. Dev was for integration tests, and DevOps had a sandbox. Client engineering and QA integrated in client, while outside vendors worked in stable. Perf ran performance tests at production scale, the client test team had test, and after that came pre-prod and prod, multi-region wherever it made sense. My favorite was demo, labeled on the board "white glove, no touching". Nothing ruins a partner demo like somebody's test running on the same servers. Releases went out blue/green, with draining and a set order, so we could switch a release over, and back again, without taking players offline.

A whiteboard sketch of environments: dev, sandbox, client, stable, demo, perf, multi-region dev, test, pre-prod and prod One environment per audience, including a demo environment nobody was allowed to touch.

None of it is real until you've rehearsed the launch. We ran full end-to-end load tests with a fake game in which two simulated players take turns, plus mock games in C# and C++ that drove the SDKs the way a real title would. We aimed for the kind of traffic a Pokémon GO event brings, and we generated enough of it to overwhelm DynamoDB in one of our regions. I'd much rather learn that in a rehearsal than on launch day.

The retrospective from one of those tests is the most honest document in this article. On the "went well" side, we found real bugs at the service level, learned how to troubleshoot across regions, ended up with a load-test system that could grow with us, improved our logging, and ran a war room that "brought us all together". The other column was just as useful. We needed clear definitions of success before starting, a proper gate for when testing begins, better test data, more involvement from the business, and a dedicated test environment. You can find that last one on the environment map above.

A retrospective whiteboard with two columns, needs improvement and went well, after a load test The retrospective after a load test. "Did it! Delivered what we set out to deliver and more."

3. The schema is the contract

A game platform serves Unity clients written in C#, backend services on the JVM and, sooner or later, consoles with their own toolchains. What held all of that together was the data contract, more than any one service. We used FlatBuffers for game data and messages. It's compact, it's fast to read on a low-end phone, and its schema compiles into every language we needed.

We used it wherever speed and throughput counted most: between services, and between the server, the game server core, the Unity engine and the mobile clients. That only works while the schema matches the server's models. Maintain it by hand next to the code and the two drift apart, and then players find the bugs before you do.

So we stopped writing it by hand. The Java models on the server became the single source of truth, and a build plugin we wrote generated the FlatBuffers schema from them. Each client language compiled its own code from that schema. The plugin started life at Pokémon, and I later open-sourced it as Aurkitu (Basque for "to find"). It's still a Maven and Gradle plugin doing the same job today. The job ended; the lesson stuck. When two systems share a contract, generate it from one place.

4. The SDK is the product

Other teams never see your architecture diagram. They see the SDK they drop into their game. We built client and server SDKs for Unity (Mono and .NET) so a game team could add accounts, purchases, matchmaking, leaderboards and game data without learning how the platform worked inside. That took days off every team's path to production, and on a launch schedule, days are what you never have enough of.

The features were rarely the hard part. The platforms underneath were. Unity's IL2CPP compiler and 64-bit builds kept us busy, every target platform needed its own test runs, and three version numbers had to be kept straight on each game server: client SDK, server SDK and game engine. Every game server reported all three through a health endpoint, so we always knew which combination was live.

That changed what "done" meant on my team. A feature was done when a game team could adopt it from the SDK and the docs alone. Working on our side was only the start. It's also why I pushed hard for thorough documentation on every service, and for a simple rhythm on every hand-off with the teams that depended on us: discuss, design, implement, test.

5. A platform team is a relationships team

The hardest problems weren't technical. A platform sits between game teams on deadlines, QA, DevOps, information security and finance, and every one of them has a fair claim on it. A few things worked for me.

Hire for range. Getting the right mix of personalities onto the team was critical. Our methodical low-level coder, with an eye for the smallest detail, was the person you want on security and the authentication and authorization paths. Another engineer was a gamer and a dad. He understood how the games would actually be played, and he became the bridge between the platform team and the people playing on it. Even the cowboy coder had his moment in the sun, turning out prototypes fast enough for us to try ideas before we committed to them.

Earn the right to standardize. We had a good relationship with DevOps and our SDETs, so they were willing to hear us out and build with us. Together we put mature deployment and testing frameworks in place, and we never had to fly the plane blind. That's also how the organization moved toward CI/CD: the platform's pipeline became the easiest way to ship, and nobody had to be told.

Give architecture a forum. I chaired the architecture team. Security, data and quality initiatives got decided there once, and stopped being re-argued on every project.

Work across companies as well as teams. Pokémon is a family of companies. The Pokémon Company itself is jointly owned by Nintendo, Game Freak and Creatures, and game work runs across studios and partners in several countries. The better you understand how each of them builds (in person, ideally), the easier every integration gets. Visiting Creatures gave us a rare look at how the card game is really made, and much of it is done by hand. Every new card and game mechanic goes through play test after play test. Along one wall ran a catalogue of the cards' whole history, every variant included. It was amazing.

Michael Hrádek beside the Creatures sign in the company's office Visiting Creatures, one of the three companies behind Pokémon.

Stay close to the code. Leading the team didn't mean handing the architecture to someone else. I stayed hands-on in Java and C#, in the services and the SDKs, and it kept my decisions honest.

Michael Hrádek in pixel sunglasses behind a laptop with Redis, AWS and IntelliJ stickers Hands-on, with the laptop stickers to prove it.

Own the budget. With an operating budget of more than $20 million, I needed to be as comfortable talking about cost per player as about requests per second.

Celebrate the games. Platform teams rarely get the spotlight, so we made a point of marking the launches around us.

A Pokémon Sword ice sculpture next to an inflatable Pikachu in the Bellevue office Celebrating Pokémon Sword and Shield in the Bellevue office.

What I'd tell anyone building a shared platform

  1. Make multi-tenancy day-one architecture. Retrofitting it later means a rewrite.
  2. Design for the spike on your worst day, rehearse it until something breaks, and decide in advance what you'll switch off.
  3. Generate the data contract from one source of truth, and look after it like a product.
  4. Measure success by adoption. Elegant services that nobody uses don't count.
  5. Spend as much time on the people around the platform as on the platform.

The Pokémon World Championships 2019 trophies, golden Pikachu figures, in a glass case The trophies at the 2019 World Championships in Washington, D.C.

The players and the games are the stars. A platform's job is never to be the story, and the best compliment a platform team can get is that nobody talks about it on launch day.