It’s easy to take the basic functionality of a social network for granted, but even something as simple as an “activity feed” (like so many apps have) has complex moving parts. And when a site scales, those parts begin to break down. For engineers at SoundCloud , rebuilding their feed technology was so labor intensive that they decided to spare the rest of humanity from the task. And thus, they opened sourced it.

Here’s their solution: It’s called Roshi, a new open source distributed storage system for “time series events” in feeds–in laymans terms, news feeds. Roshi was developed by Peter Bourgon and other engineers at SoundCloud in order to scale up without slowing down the performance of the very social streams that keep users engaged.

Let’s say you follow Snoop Dogg on SoundCloud. If he reposts (SoundCloud’s equivalent of retweeting) a track by an up-and-coming hip hop artist, you’d naturally expect to see the song in your stream. The traditional way of doing this, known as “fan out on write” essentially treats each user’s stream (equivalent to the News Feed on Facebook) as an inbox, pushing updates out to each of them individually. It works, but it’s inefficient, the storage costs add up, and changes to the social graph can become a pain to implement.

“At some point, those caveats and restrictions started affecting our ability to iterate on the stream,” explains Bourgon. “To keep up with product ideas, we needed to address the infrastructure. And rather than tackling each problem in isolation, we thought about changing the model.”

SoundCloud’s new approach relies on a methodology called “fan-in-on-read.” When you view the stream of SoundCloud users you follow, the system will grab the most recent events (favorites, reposts, and the like) of those people and then dynamically merge that information on the fly. It speeds up writes and minimizes storage, but presents new challenges.

“Reads are difficult,” Bourgon explains. “If you follow thousands of users, making thousands of simultaneous reads, time-sorting, merging, and cutting within a typical request-response deadline isn’t trivial. As far as we know, nobody operating at our scale builds timelines via fan-in-on-read.”

For Bourgon and the other developers working on Roshi, the solution came in the form of a specific CRDT (Convergent Replicated Data Type). These conflict-free replicated data types “manage to sidestep a lot of the common problems and pitfalls associated with distributed systems,” Bourgon explains.