Liz: Today we’ve got a guest post from the terrifyingly hirsute Pete Stevens. Pete’s from Mythic Beasts, our web hosts; and he’s the reason this website stands up to the absurd amounts of traffic you throw at it. (Yesterday we saw about a quarter of a million sessions – that goes up WAY above a million on some days.) Have at it, Pete!

After our successful test of using the Raspberry Pi 3 for hosting 5% of the traffic on Pi 3 launch day we celebrated by going to the pub. The conversation went something like this:

Eben: Is it possible to host the whole site on Pi 3s?

Pete: How would you do it?

Philip: Wouldn’t it be awesome to do it?

Liz: Dare you to try it!

The first part of the answer is quite easy: not on one Pi 3, it’s not fast enough. A better question is how many Pi 3s are required to host a typical day’s traffic. Extrapolating from some graphs and making up some numbers with the handy pub beermat service, we estimated between 4 and 6 should handle all the PHP code and file delivery. Unfortunately, the database server still looks out of scope: not enough RAM and not enough I/O.

Of course, only an idiot would replace thousands of pounds of highly specified hardware with a handful of £30 computers and expect it to still work.

A few weeks later we have this:

We’ve designed a custom plastic enclosure for holding Pi 3s securely, added power over ethernet HATs so we can power them directly from the switch, and a cheap 100Mbps PoE switch. We’ve put all the storage over the network and put a small storage server in the rack with the Pi 3 rack. We’ve used virtual LANs to have two effective network cards on each Pi 3: one just containing it and the storage server, the other with an IPv6 address that talks to the public internet and the load balancers. Ifconfig looks like this:

storage : eth0 : 10.46.189.X public : eth0.131 : 2a00:1098:0:84:1000:1:0:X

As with all Pi servers, there is no public IPv4 connectivity to each server. To get out to legacy IPv4-only services such as Twitter / Akismet etc. they go through our NAT64 DNS proxy service. Inbound traffic lands on the front end load balancers and is shared between the Raspberry Pi 3s over IPv6.

If you do:

HEAD -E https://www.raspberrypi.org/

you’ll see a header which gives you the final octet of the address of the Pi that served you:

X-Served-By: Raspberry Pi 1e

The first person to tweet all the hex identifiers to Mythic Beasts wins absolutely nothing other than the respect of the Raspberry Pi community.

Is this a commercial hosted Pi service?

It’s not yet a commercially viable service. Scaled up we can fit 96 Pi3s in 4U of rack space including the switches, which is an impressive density. However, the Pis aren’t individually replaceable once in service. That means if a customer botches the SD card the Pi is dead until we can arrange downtime of all 96 Pis in the unit. Kernel upgrades involve a change on the SD card which carries a risk of bricking the Pi if the user gets it wrong. Not having access to the SD card other than via booting the Pi from it means that an enterprising user could compromise the kernel on the SD card and root-kit the machine, before cancelling the service and letting us sell it to another user.

But it’s close. Add in netboot with PXE and most of the above concerns go away, as we can remotely provision, remotely re-provision and remotely recover a broken Pi.