The Magic Mirror that I’ve got has failed a bunch of times. Each time, the SD card has become corrupt and the mirror then hangs accusingly on the wall waiting for me to repair it. I’ve hidden away the keyboard. I need to unhang the mirror to access the SD card, and each time the wonderful mounting (a.k.a. masking tape) that holds the pi perfectly in place all needs to be redone. This is a pain. I finally decided to sit down and work out a way of managing the pi remotely – to complete ditch these pesky SD cards.

The end result? My mirror now has a raspberry PI v3, running without SD card, completely running from the network. To mess with the boot settings or the filesystem, I can completely do this from my comfortable workstation, remotely reboot the pi and it’s been pretty reliable (not perfect). The network service is all provided by my iMac, all via OSX without any software other than what’s built-in, or reputable open-source software. This setup is so much more comfortable to use.

I’ll walk through the steps to get this configuration working, but first a little background on the devices at play and what we’ll need to mess with. I’ve got a fairly typical setup: broadband arrives via a router from my provider (FIOS in my case). The default configuration is where I’m starting from, and I’m not going to reconfigure my entire home network (that’s phase 2), just enough to make this specific project work. But that does require to go into the router configuration to change the settings. On the home network, the two machines of relevance here are the raspberrypi and the OSX machine. I’m using a modern iMac with High Sierra installed (OSX 10.13.4). The raspberry pi is a pi3B. Key things to note here:

The raspberry pi is v3. I have a pi2 around somewhere, but from commentary on how recently network booting has been solid, decided to go to the pi3 for this config. I’m just using raspbian stretch. I’m tempted to play with some container builds, but there are reports that the stacked filesystem layers don’t work with NFS mounts, so I’ll leave that for a later experiment.

The pi and the iMac are both using physical networking rather than wifi. Potentially the iMac acting as the server could be wifi, but it felt to me that I wanted far greater control and less unknown factors over what was happening, so keeping the server and the client on the same wired side of the switch made sense.

I’m assuming you’ve already got Homebrew installed on the OSX server in order to easily grab open source packages.

Step 1: Get the PI network boot ready

At some point, we need to use an SD card, and this is it. The beginning part of the standard documentation Just Works, up until “server configuration” which is where the standard docs starts to fail us. Feel free to get the pi as configured as you want – clean up installed software, fixup hostname/users, timezone, etc. Doesn’t matter too much to be honest when you do this kind of admin – do it now, or do it later when you’re networked boot. Same end-result. Get the pi ready to go, get the network boot enabled (program_usb_boot_mode), shutdown the pi and grab that SD card and stick it into the OSX machine. You don’t need the SD card in the pi ever again!

Step 2: Configure the Network

We’ve got the pi and the OSX server hardwired ethernet, but the default configuration won’t work for us. We will need to have the OSX server on a static IP address (otherwise NFS will get really confused), and to simplify things we’ll do the same for the pi. If your home network is anything standard, then the broadband access point will be providing dynamic IP address for the home network and we need to mess with that.

First step is to just write down what your current config is so that you know how to fix things later! Go into System Preferences on the OSX host and look at the Network tab. Write down the IP address, subnet mask, router, DNS server and search domains. As an example, my IP address was 192.168.1.222, with a subnet mask of 255.255.255.0.

I can also pull up the configuration on the broadband router and see that its configuration (Advanced/IP Address Distribution on a FIOS modem) says that it’s got a dynamic range for 2-254 for the host part of the address. We’re all consistent, good.

I decided to pick 192.168.1.2 for my OSX server address and to use 192.168.1.3 for the raspberrypi. I’ll probably have more static addresses that I’ll want to use, so in the router configuration, I changed the address distribution to only give out addresses from 10-254. That gives us a small pool of static space (2-9) that I can assign manually.

Back into System Preferences and change the configuration to Manual and fill in the information we’ve just worked out. The IP address must be one in the static range we just decided (in this case 192.168.1.2), with all the other fields being the same as they were when the configuration was managed by DHCP from the broadband router. When you do this, you have to click the advanced tab, so that you can also assign the DNS server (from the settings noted down previously). I don’t know why DNS server is advanced but a subnet mask is not, but hey, I’m not a UX designer. My OSX network screen now looks like this:

Just check at this point that your network still works – pull up a website, browse around, everything should still be good. If anything isn’t right, then stop and fix. There’s not a lot that can go wrong at this point to be honest. Should all be fine (famous last words).

Step 3: Provide a filesystem for the pi via NFS

IMPORTANT NOTE: The pi expects a filesystem that has case sensitive filenames. The default filesystem under OSX is case insensitive. Make a new partition that is case-sensitive and use that for the pi. If you don’t, everything works… everything looks fine… until one day you get hit by some weird error during a kernel upgrade. Grrr…. Remember to also do “Get Info” from Finder on the new partition, and uncheck the strange option “ignore ownership” which will probably be set, that will also confuse things!

We need to get the content of the filesystem that was constructed when you first built the pi, and make it available via the network. There is a built-in NFS server in OSX (activated when you click on the “file sharing” in System Preferences/Sharing tab). Getting the content of the pi filesystem is a little more complicated. When you mount the SD card on the OSX host, it only shows the boot partition, because OSX doesn’t know about Linux filesystems, so you need to do a little bit of magic here. I followed the information from Jeff Geerling’s blog about ext4fuse that worked like a charm. If you follow that guide, you’ll be able to see the root filesystem on the SD card, and you can copy the content into somewhere persistent on the OSX host. I decided to store this under /exports/pi-root (for no good reason, to be honest, I may move that somewhere better later). The following commands create that directory and make it available to NFS. Note: the rsync takes about 5 minutes to complete, depending on the speed of your SD card.

osx$ sudo mkdir -p /exports/pi-root osx$ sudo rsync -a /Volumes/rpi/ /exports/pi-root/ osx$ sudo echo "/exports/pi-root -maproot=0:0 -network 192.168.1.0 -mask 255.255.255.0" >> /etc/exports osx$ sudo service stop nfsd osx$ sud o service start nfsd

There’s a problem here (I’m interested if folk have ideas to fix this). The pi was (I assume) using ACLs on some directories, which will not have been copied across, nor will they be available to NFS. So a couple of directories will need to be opened up. Notably, I saw that I needed:

osx$ sudo chmod 755 /exports/pi-root/home/pi

After this, you should have /exports/pi-root all ready to go with the content of the root filesystem of the pi.

Step 4: Get the Server providing DHCP (pxeboot)

Okay, let’s start trying out some DHCP and make sure that the OSX server and the pi can see each other and we have a chance this might work. Even before we’ve configured any services, we should still be able to see the pi asking for network boot. We want to turn on the pi while snooping the network and confirm we see DHCPREQUEST messages on the OSX machine. So, on the OSX server, go to a command line and run “sudo tcpdump -i en0 port bootpc”. Note that I’m using en0 as my network interface to listen on, because that’s the wired interface I’m on. Your mileage may vary, you may be somewhere else. Easy way to check this is to look at “ifconfig -a” and look for the IP address that the host was just given. For example “ifconfig -a | grep -4 192.168.1.2”. Here’s what I see when the pi is powered on:

osx$ sudo tcpdump -i en0 port bootpc listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes 14:48:31.869444 IP localhost.bootpc > broadcast.bootps: BOOTP/DHCP, Request from b8:27:eb:8a:53:3e (oui Unknown), length 320 ^C

There will be five DHCP requests coming from the pi before it eventually gives up. If you want to try again, you need to power cycle the pi in order for it to start again. Take a note of the ethernet address here (Request from b8:27:…), because you need to use that ethernet address within the DHCP config a little later.

If you don’t see any traffic going by, then you need to debug the network config to fix it. If you can’t see this traffic, this isn’t going to work. If you do see the DHCP requests, then yay! we’re on to a winner and can proceed.

We need to have a server that responds to the DHCP requests with the right information to allow a pi to network boot. To my mind, the easiest way of doing this is just to fire up ISC-DHCP, since it’s pretty easy to configure for this kind of stuff. You just need to:

osx$ brew install isc-dhcp

Of course, we need a DHCP configuration. Assuming the previous network definitions I’ve used, create /usr/local/etc/dhcpd.conf with content like the below. Note the reference to the NFS location of the pi’s root filesystem. You will want to change your server-name and domain-name based on what you grabbed earlier. Also, you want to put the raspberrypi ethernet address you grabbed from the tcpdump a few minutes earlier. Basically change the highlighted values.

# ISC-DHCP server configuration authoritative; ddns-update-style none; default-lease-time 43200; max-lease-time 86400; server-name " alpha.fios-router.home "; # DNS servers that your clients will use option domain-name-servers 192.168.1.1; # Search domain that your clients will use option domain-name " fios-router.home "; subnet 192.168.1.0 netmask 255.255.255.0 { option broadcast-address 192.168.1.255; option subnet-mask 255.255.255.0; option routers 192.168.1.1; } host net-booting-pi { option root-path "192.168.1.2:/exports/pi-root,tcp,vers=3"; option vendor-encapsulated-options "Raspberry Pi Boot "; next-server 192.168.1.1; hardware ethernet b8:27:eb:8a:53:3e ; fixed-address 192.168.1.3; }

You should now be able to test DHCP config on your OSX host:

osx$ sudo /usr/local/sbin/dhcpd -t -f Internet Systems Consortium DHCP Server 4.4.1 Copyright 2004-2018 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Config file: /usr/local/etc/dhcpd.conf Database file: /usr/local/var/dhcpd/dhcpd.leases PID file: /usr/local/var/dhcpd/dhcpd.pid

Okay, so looking good! Fire up dhcpd without test mode, by just using “dhcpd -d” (we want the -d option to see debug messages), and try again power-cycling the raspberry pi. We would hope that the DHCP server would start telling us that it sees the DHCP requests and is trying to do something about it! If everything is good at this point, then you’ll see a single DHCP request being made by the pi, followed by a DHCPOFFER from OSX’s dhcpd. If you see the offer and the DHCP server replying, but after a few seconds the raspberrypi keeps asking again (up until its usual five times), it means that the response wasn’t up to snuff. The most likely culprit here is the “vendor-encapsulated-options” isn’t right. Go check that.

A final bit of cleanup. Make dhcpd persistently running using:

osx$ sudo brew services start isc-dhcp

So, if we’ve got DHCP working, then it means that the pi will now be asking for its boot images. You can validate this if you want by checking the output of tcpdump. You want something like “sudo tcpdump -i en0 host ether $ethernet-address-of-the-pi”, so that you can see all traffic with the pi. You should see something like:

osx$ sudo tcpdump -i en0 ether host b8:27:eb:8a:53:3e tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes 16:12:02.824101 IP localhost.bootpc > broadcast.bootps: BOOTP/DHCP, Request from b8:27:eb:8a:53:3e (oui Unknown), length 320 16:12:02.824370 IP 192.168.1.2.bootps > 192.168.1.3.bootpc: BOOTP/DHCP, Reply, length 300 16:12:03.850496 ARP, Request who-has 192.168.1.2 tell 192.168.1.3, length 46 16:12:03.850572 ARP, Reply 192.168.1.2 is-at d0:81:7a:da:24:b0 (oui Unknown), length 28 16:12:03.850704 IP 192.168.1.3.49152 > 192.168.1.2.tftp: 21 RRQ "bootcode.bin" octet ^C

Step 5: Get the Server providing TFTP boot images

A TFTP server is included in OSX, although it is not running by default. So, we just get it configured to run, which is pretty easy:

osx$ sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist

For some reason, debugging output from tftpd I couldn’t get working. I tried modifying the tftp.plist (by copying a new labelled tftp service, to get around OSX SIP and the like), but even so didn’t get any useful logging. So, just use the standard configuration and rely on simple tcpdump to test that it works. Each time I’m messing with one of these services, I’m just running ‘tcpdump’ in the same way as before, to see what the pi is doing on the network.

Where do we get the TFTP files from? This is where the SD card comes back into play. The SD card should still be mounted in the OSX host, and it will show the /boot volume, which contains everything the TFTP server needs to provide. So, easiest thing to do here is to just copy all of that data into the TFTP area

osx$ sudo rsync -av /Volumes/boot/ /private/tftpboot/

Step 6: Almost There!

There are a bunch of files that need to be modified to make the network boot successful. We have config.txt and cmdline.txt now coming from TFTP so we need to edit those in /private/tftpboot, and there are a couple of files on the root filesystem that we need to edit in /exports/pi-root/. Notably, you need to edit cmdline.txt to say that the root filesystem comes from NFS and is read/write (don’t need the actual nfsroot details, since they will come from DHCP), and you need to edit /etc/fstab to say that the filesystems do not come from the local SD card.

osx$ cat /private/tftpboot/cmdline.txt dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs rw ip=dhcp rootwait elevator=deadline net.ifnames=0 osx$ cat /exports/pi-root/etc/fstab proc /proc proc defaults 0 0 # a swapfile is not a swap partition, no line here # use dphys-swapfile swap[on|off] for that

Try power cycling the pi at this point and this should look good. The pi gets a DHCP offer, starts asking for TFTP data and there’s a lot of files being downloaded (seen through tcpdump). It should then mount the root filesystem and start booting normally.

Things that could go wrong here? I hadn’t originally fixed /etc/exports and so the boot process got very upset and failed. Easy to fix (and spot, since you see references to a disk that is no longer there!)

I had to spend a while debugging things and it turned out that rootsquash was causing a load of problems. Unsquashed that (the maproot line in /etc/exports on the OSX host), and now the system works, and I can login.

It took me a while to login and I’m not sure if that was while the filesystem was readonly (because of root squash), or if it was because of SSH keys, but just in case, I removed the old host keys in /etc/ssh and edited etc/rc.local to run dpkg-reconfigure openssh-server to make sure that the SSH keys were regenerated. All of this is easy to do when you can just go into /exports/pi-root and edit whatever you want.

Okay, so now the host is on the network, running off NFS, and I can ssh in. Muchos better! Weirdly, X didn’t do the right thing – the session started, but nothing displayed on the screen. Not sure what’s happening there.

Okay, logged in, we can see in the logs that the X sessions are constantly restarting. lightdm is trying to create/update .Xauthority for the user, but it cannot write to the file. The file was marked with mode 0400, so even the owner could not write to the file. Modified it so that it was 0600 and now everything works. Possibly an ACL at play in ext4, that doesn’t translate to NFS? If so, are there other gotchas like that…?

After a long time of everything working fine, installing new software, editing, all good, eventually I hit a weird error where the raspberrypi-kernel would not update (apt-get install would give strange errors about “File Exists”, or “No such file” errors). It was really strange and took some time to realize that there are files in the kernel modules that differ only by case, which meant that the underlying OSX filesystem that defaults to case-insensitivity was giving false positives on file existence. So, I created a new case-sensitive partition and moved all the NFS exported filesystem over there and everything was then happy for a kernel upgrade.

Extra Thoughts

I’m running the NFS server allowing root access to the filesystem. I don’t like that, but it’s not going to work read/write without a lot of other messing around. Notably, things like /var/log will be written to by root. Sure you can make /var a ramdisk, but it would be better to configure things like remote syslog service so that we can usefully debug problems. Cleaning up that kind of config is for another day…

rasps-config tries to edit /boot/config.txt which doesn’t work, because /boot on the pi is not pointing to the right place. Instead, you need to manually make changes in osx:/private/tftpboot/config.txt. I’m not sure of a clean way of fixing this, except maybe mounting that up as another filesystem through the NFS server. I’m just ignoring that problem for now.

I’ve had one repeatable OSX crash since I’ve been running this. Every time I try to run a particular ‘dpkg’ install command, the OSX host will panic. I’m still trying to work out what’s happening there. Until I work it out, I’m just not running that command :).

I’ve had the pi fail to boot a couple of times – it just hangs on the rainbow display. something like 2 out of 20 times. Not sure what’s up there.

Raspbian panics if I try to NFS boot and also load the kernel module bcm2835-v4l2. Don’t do that.