Playing League of Legends for years now, I’ve formed a meaningful network of social connections with other gamers around the world. Whether they’re friends from work, former classmates, or players I’ve been matchmade with, they all have an important place on my friends list. The ability to easily play with these friends greatly enhances my experience with the game. It would be disastrous if something ever happened to that social graph—trying to remember and re-friend all of my 200+ friends would be as bad as losing my phone and all of its stored contacts.

I trust that Riot chat servers will persist information about my account—for example, the roster of my friends, metadata about a certain buddy, or the list of players that I have blocked. Furthermore, that data should be available to me at any time, anywhere, and stored securely so that only I can access it. In this article, I’ll walk through a brief history of our chat persistence layer and also outline the architecture that powers it today. I hope anyone facing a technical storage and persistence problem will find this post useful and interesting.

Early MySQL days

A few years ago, before the size of the LoL player base really started taking off, we made the decision to build chat servers using an open source XMPP server implementation: ejabberd . (You can read more about the protocol and server-side implementation in previous posts of mine.) In the past ejabberd offered only a few persistence layers: you could either use mnesia or an ODBC backend (which meant MSSQL, PostgreSQL, or MySQL servers).

We chose MySQL as the main store for chat data because scale wasn’t our primary concern at the time, and we had a lot of in-house experience with it. The implication was that all data that needed to be stored on the server and persisted between player logins had to be saved to MySQL. This included:

Friends lists (aka rosters) and metadata about your friends such as notes, groups, friendship status, creation date, etc.

Privacy lists: a register of blocked players whom you don’t want to hear from.

Offline messages: all chat messages that were sent to you while you were offline, and which should be delivered as soon as you log back in.

We built each shard to have three MySQL servers: one instance each for master, backup, and ETL (Extract, Transform, Load). The master is responsible for handling reads and writes from the chat servers and replicating its data to both the backup and ETL servers. The backup instance is there to (surprise!) snapshot data and be ready to act as the master server in the case of either maintenance or failure of the original master. Finally, the ETL server is used for data analysis and can be deployed on slightly less performant hardware.

Overall, chat shards using MySQL as a persistency backend look like this: