If you play games on PC, where God intended them to be played, chances are you’ve got Steam installed. Since its troubled launch in 2003, Valve’s publishing platform has gone from a thing we had to grudgingly put up with in order to play Half Life 2 to the most popular digital game distribution tool on the planet. Most of us—me included—interact with it pretty much daily.

But as game distribution was shifting from physical to digital, ISPs also began implementing data caps—usually under the guise of “network management” (though anyone who thinks caps aren’t a pure revenue play should send me an e-mail, because I’ve got a bridge to sell you, cheap!). Steam is fast, Steam is easy, and Steam is ubiquitous—but if you’ve just rebuilt a PC or reinstalled an OS and you need to reinstall your games, Steam will obligingly help you put a giant dent in your cap—very quickly.

But there’s an alternative to having to re-download all your Steam games from the Internet: you can set up a local Steam caching server, so that once you download something, you’ve got it on your LAN instead of having to reach for it across the net and incur usage fees.

How this works

The idea is simple: when your computer’s Steam client needs to install or update an application, it contacts one or more of Valve’s SteamPipe content distribution servers. Valve has partnered with ISPs across the world so that most folks have SteamPipe content servers near them (“near” in terms of both physical distance and also network hops). This saves ISPs money on transit and peering.

SteamPipe is used to deliver what the client needs, be it a whole game or just an update, in roughly megabyte-size chunks. (Chunking like this allows developers to publish updates without having to push a whole new game package—they just invalidate old chunks and upload new ones.) As Valve points out on the SteamPipe developer community page, SteamPipe uses plain ol’ HTTP rather than a proprietary protocol. And that gives us the opportunity to stick our fingers into the process and mess with it.

Recall for a moment how a caching Web server works: a user hits a page, and the server checks its cache to see if the cache has what the user needs in it. If the cache does, the server delivers those objects directly. If not, the caching Web server forwards the request to a backend, retrieves whatever the user needs from that backend, and delivers it. Then the Web server stores the objects in cache for next time so it doesn’t have to bother the backend again.

What if we turn that process sort of inside out? What if we set up a caching Web server like Nginx locally? What if, instead of using the server as a reverse-proxy to cache responses for incoming requests from the Internet for a particular Web site... we use it as forward proxy (or just a regular proxy, I suppose) and cache the responses to our outgoing requests for Steam content?

It sounds like we’re inverting what a reverse-proxying Web server is supposed to be for and using it like a normal Squid-style proxy—but as it turns out, this actually works great.



Did I say “great”? I meant “almost great.” There’s another piece of the puzzle. We need to screw around a bit with DNS for this to actually work. Steam doesn’t have a “check my local server first” option to make this caching thing an automated process, so we’ll need to add DNS entries to fake Steam into thinking that its SteamPipe content server actually is the local cache server we’re going to build.

All roads lead to Rome, but this is our road

Fortunately, none of this is actually as complicated as it sounds—and, as we’ll see, there are pre-built Docker images to do all of this for you, if you don’t want to get your hands dirty.

But before we proceed any farther, we need to go over exactly what you need for your local Steam cache server. As with so many projects, you can take a huge variety of paths to get to the destination; you can do this with nothing more than Windows and your gaming PC, or you can use a Windows or Linux virtual machine, or you can use an actual physical Windows or Linux server. You can keep your Steam cache on local disk or on a network share. You can use whatever application you want, provided it can forward and cache HTTP requests.

For this guide, we’re going to look at two specific configurations: Linux (specifically Ubuntu 16.04 LTS Server) with Docker for the easy way; and Linux (again, Ubuntu 16.04 LTS Server) on bare metal for the DIY way. Both configurations will store the actual cache depot on a network share—I set my depot up using iSCSI for a number of reasons external to this guide, but there’s nothing wrong with NFS (or even Samba, I guess). If you have a few terabytes of disk space, you can also put your depot local on your server. It’s all up to you and what works for you, but this guide will talk about putting your depot on a NAS and accessing it via a mounted NFS share.

The star of our show in both configurations is going to be Nginx, a fast and powerful Web server that runs a significant chunk of the World Wide Web (it’s second in popularity only to Apache, the Microsoft Word of Web servers). Nginx’s ability to maintain its own local content cache from an upstream source is what makes this entire process work. If Nginx sounds familiar to you, it’s because we’ve written about using it at great length with our Web Served series (and, yes, those articles are in dire need of update—that’s on my 2017 to-do list!).

Other tools could be employed here, too—most notably, a standard proxy server like Squid. And, in fact, guides are out there that tell you how to configure Squid for just that. But we’re going with Nginx primarily because of all the documentation out there to do it this way, and also because there are Docker images of working configurations we can pick apart and use.