FreeBSD on EdgeRouter Lite - no serial port required

I recently bought an EdgeRouter Lite to use as a network gateway; I had been using a cheap consumer wifi/NAT device, but I wanted the extra control I could get by running FreeBSD rather than whatever mangled version of Linux the device came with. Someone wrote instructions on installing FreeBSD onto the EdgeRouter Lite two years ago, but they rely on using the serial port to reconfigure the boot loader — perfectly straightforward if you have a serial cable and know what you're doing, but I decided to take the opportunity to provide a more streamlined process.

Required:

An EdgeRouter Lite.

A small Phillips ("plus") screwdriver.

Another system running FreeBSD.

Steps:

Fetch a FreeBSD HEAD source tree. (FreeBSD 10-STABLE is not supported yet. I think this might change between now and 10.3-RELEASE.) Download the image building script. Run ./buildimg.sh /path/to/src/tree disk.img . Remove three small screws from the back of the EdgeRouter Lite. Open the case and remove the USB drive. (Mine was held very firmly in place. I found that wiggling it towards and away from the board allowed me to gradually ease it free.) Plug the USB disk into the system where you built the FreeBSD image. Run dd if=/dev/USBDISK of=ERL.img where USBDISK is the name of the USB disk device (probably da0 ), to make a backup of the EdgeRouter Lite software in case something breaks and you need to restore it later. Run dd if=disk.img of=/dev/USBDISK (where USBDISK is as before) to write the FreeBSD disk image onto the EdgeRouter Lite USB disk. Plug the USB disk back into the EdgeRouter Lite, close the box, and replace the three screws.

There are three gigabit ethernet ports on the EdgeRouter Lite, marked on the case as "eth0", "eth1", and "eth2"; in FreeBSD, they show up as "octe0", "octe1", and "octe2" in the same order. With the configuration on my image:

"eth0" should be connected to your upstream internet connection; the system runs DHCP on this connection to get an IP address and sends internet traffic out this port.

"eth1" should be connected to your LAN; the system provides IP addresses within the 192.168.1.0/24 block via DHCP, and runs a recursive DNS server (also advertised via DHCP) and an NTP server on 192.168.1.1.

"eth2" is like "eth1", except that it gets IP addresses from the 192.168.2.0/24 block and traffic from this port (and only this port) can connect via SSH using the username and password "ubnt". (I recommend leaving this port unused except when you need to SSH in to reconfigure something.)

That's pretty much all you need to know to install and use FreeBSD on the EdgeRouter Lite; but there are some interesting tricks involved in the script which builds the disk image, so for the rest of this blog post I will provide a brief "walkthrough" of the script.

#!/bin/sh -e

/bin/sh

/bin/bash

-e

if [ -z "$1" ] || [ -z "$2" ]; then echo "buildimg.sh srcdir disk.img" exit 1 fi SRCDIR=$1 IMGFILE=$2

# Set environment variables so make can use them. export TARGET=mips export TARGET_ARCH=mips64 export KERNCONF=ERL

export ALL_MODULES=YES export WITHOUT_MODULES="cxgbe mwlfw netfpga10g otusfw ralfw usb rtwnfw"

pf

# Create working space WORKDIR=`env TMPDIR=\`pwd\` mktemp -d -t ERLBUILD`

# Build MIPS64 world and ERL kernel JN=`sysctl -n hw.ncpu` ( cd $SRCDIR && make buildworld -j${JN} ) ( cd $SRCDIR && make buildkernel -j${JN} )

-j

make

sysctl

# Install into a temporary tree mkdir ${WORKDIR}/tree ( cd $SRCDIR && make installworld distribution installkernel DESTDIR=${WORKDIR}/tree )

installworld

installkernel

distribution

# Download packages cp /etc/resolv.conf ${WORKDIR}/tree/etc/ pkg -c ${WORKDIR}/tree install -Fy pkg djbdns isc-dhcp43-server rm ${WORKDIR}/tree/etc/resolv.conf

# FreeBSD configuration cat > ${WORKDIR}/tree/etc/rc.conf <<EOF

/etc/rc.conf

hostname="ERL"

growfs_enable="YES"

tmpfs="YES" tmpsize="50M"

/tmp

ifconfig_octe0="DHCP" ifconfig_octe1="192.168.1.1 netmask 255.255.255.0" ifconfig_octe2="192.168.2.1 netmask 255.255.255.0"

pf_enable="YES" gateway_enable="YES"

sendmail_enable="NONE" sshd_enable="YES" ntpd_enable="YES" ntpd_sync_on_start="YES" svscan_enable="YES" dhcpd_enable="YES" dhcpd_ifaces="octe1 octe2" EOF

cat > ${WORKDIR}/tree/etc/pf.conf <<EOF # Allow anything on loopback set skip on lo0 # Scrub all incoming traffic scrub in # NAT outgoing traffic nat on octe0 inet from { octe1:network, octe2:network } to any -> (octe0:0) # Reject anything with spoofed addresses antispoof quick for { octe1, octe2, lo0 } inet # Default to blocking incoming traffic but allowing outgoing traffic block all pass out all # Allow LAN to access the rest of the world pass in on { octe1, octe2 } from any to any block in on { octe1, octe2 } from any to self # Allow LAN to ping us pass in on { octe1, octe2 } inet proto icmp to self icmp-type echoreq # Allow LAN to access DNS, DHCP, and NTP pass in on { octe1, octe2 } proto udp to self port { 53, 67, 123 } pass in on { octe1, octe2 } proto tcp to self port 53 # Allow octe2 to access SSH pass in on octe2 proto tcp to self port 22 EOF

octe2

mkdir -p ${WORKDIR}/tree/usr/local/etc cat > ${WORKDIR}/tree/usr/local/etc/dhcpd.conf <<EOF option domain-name "localdomain"; subnet 192.168.1.0 netmask 255.255.255.0 { range 192.168.1.2 192.168.1.254; option routers 192.168.1.1; option domain-name-servers 192.168.1.1; } subnet 192.168.2.0 netmask 255.255.255.0 { range 192.168.2.2 192.168.2.254; option routers 192.168.2.1; option domain-name-servers 192.168.2.1; } EOF

# Script to complete setup once we're running on the right hardware mkdir -p ${WORKDIR}/tree/usr/local/etc/rc.d cat > ${WORKDIR}/tree/usr/local/etc/rc.d/ERL <<'EOF' #!/bin/sh

EOF

# KEYWORD: firstboot

/etc/rc

# PROVIDE: ERL # REQUIRE: NETWORKING # BEFORE: LOGIN # This script completes the configuration of EdgeRouter Lite systems. It # is only included in those images, and so is enabled by default. . /etc/rc.subr : ${ERL_enable:="YES"} name="ERL" rcvar="ERL_enable" start_cmd="ERL_run" stop_cmd=":"

ERL_run() { # Packages env SIGNATURE_TYPE=NONE pkg add -f /var/cache/pkg/pkg-*.txz pkg install -Uy djbdns isc-dhcp43-server

# DNS setup pw user add dnscache -u 184 -d /nonexistent -s /usr/sbin/nologin pw user add dnslog -u 186 -d /nonexistent -s /usr/sbin/nologin mkdir /var/service /usr/local/bin/dnscache-conf dnscache dnslog /var/service/dnscache 0.0.0.0 touch /var/service/dnscache/root/ip/192.168

dnscache

# Create ubnt user echo ubnt | pw user add ubnt -m -G wheel -h 0

-h 0

# We need to reboot so that services will be started touch ${firstboot_sentinel}-reboot

} load_rc_config $name run_rc_command "$1" EOF chmod 755 ${WORKDIR}/tree/usr/local/etc/rc.d/ERL

# We want to run firstboot scripts touch ${WORKDIR}/tree/firstboot

/firstboot

/etc/rc

# Create FAT32 filesystem to hold the kernel newfs_msdos -C 33M -F 32 -c 1 -S 512 ${WORKDIR}/FAT32.img mddev=`mdconfig -f ${WORKDIR}/FAT32.img` mkdir ${WORKDIR}/FAT32 mount -t msdosfs /dev/${mddev} ${WORKDIR}/FAT32 cp ${WORKDIR}/tree/boot/kernel/kernel ${WORKDIR}/FAT32/vmlinux.64 umount /dev/${mddev} rmdir ${WORKDIR}/FAT32 mdconfig -d -u ${mddev}

/vmlinux.64

# Create UFS filesystem echo "/dev/da0s2a / ufs rw 1 1" > ${WORKDIR}/tree/etc/fstab makefs -f 16384 -B big -s 920m ${WORKDIR}/UFS.img ${WORKDIR}/tree

makefs

makefs

# Create complete disk image mkimg -s mbr \ -p fat32:=${WORKDIR}/FAT32.img \ -p "freebsd:-mkimg -s bsd -p freebsd-ufs:=${WORKDIR}/UFS.img" \ -o ${IMGFILE}

da0s2a

# Clean up chflags -R noschg ${WORKDIR} rm -r ${WORKDIR}

Shell scripts are run by. (No matter what some misguided Linux users may think,is not a standard shell.) Theoption tells the shell interpreter to exit if any of the commands fail — if something goes wrong, we should stop and let the user see what happened rather than continuing and producing a broken disk image later.This script takes two options: The location of the FreeBSD source tree, and the name of the file to use for the disk image.The EdgeRouter Lite is a 64-bit MIPS system; FreeBSD HEAD has a kernel configuration already defined for it.Unfortunately that kernel configuration disables building kernel modules; we want those so that we can uselater. Some modules fail to build — I didn't investigate exactly why, but it was something related to firmware blobs — so we turn those off explicitly.We create some temporary working space under the current directory. On many systems /tmp isn't large enough to hold a complete installation of FreeBSD, so I overrode the default there.Build the MIPS64 world and kernel. Theflag tellsto run several commands in parallel; we consultto find out how many CPUs we have available for the build.We create a tree and install FreeBSD into it. Theandtargets install the userspace and kernel binaries respectively; thetarget installs standard configuration files.The FreeBSD project provides precompiled binary packages for the 64-bit MIPS architecture; this allows us to put packages into the image we're building while avoiding the headaches of cross-building them. However, we cannotpackages either, since packages can run scripts when they are installed — scripts which (since we're not building this disk image on a MIPS64 system) we won't be able to run. Instead, we simply download the packages into the image; they will be installed when the system first boots.Thefile is the "master configuration file" on FreeBSD; most enabling/disabling of services is done here, as well as some more specific configuration.Every host needs a name. We'll call this "ERL", lacking any better inspiration.We're building a disk image which we'll write onto the provided USB disk, but the image is smaller than the disk; when the system first boots, this tells it to expand the root partition to fill the available space.This is probably unnecessary, but I like to have a memory disk mounted on; if for some reason temporary files get created here, this will avoid burning up the flash storage.We run DHCP on the "upstream" connection, but provide static network parameters for the "LAN" connections.We're going to use the PF firewall; and we're going to be forwarding packets (both via the network address translation and between the two LAN ports) so we need that option too.We don't want to run sendmail; we do want to run sshd (we'll use PF to restrict access, however); we do want to run ntpd, and we want it to set its clock when it starts, no matter how far off it is (the EdgeRouter Lite doesn't have a battery-powered clock, so it boots with a wildly wrong time set); we want to run svscan so that it can launch dnscache for us; and we want to run a dhcp daemon for the two LAN interfaces.Fairly straightforward PF configuration: NAT outgoing traffic onto the "upstream" connection; allow the local network to access DNS, DHCP, and NTP; and allowto access SSH. I opted to only allow ICMP echo request packets from the LAN side — some people prefer to respond to pings from anywhere, but I decided that for a general purpose image it was better to err in the direction of being silent. Similarly I decided to simply drop bad packets rather than sending TCP RST or ICMP unreachable responses.This provides a basic configuration for ISC DHCPD. I have a feeling that this could be simplified to have a single configuration block covering both LAN ports.I mentioned earlier that we couldn't cross-install packages; we take care of that now, with a script which runs the first time FreeBSD boots. The quotes aroundin the here-document syntax instruct the shell that variables should not be expanded — important since we're creating a shell script which uses several shell variables.The "firstboot" keyword tellsthat this script should only be run the first time that the system boots.This is fairly standard rc.d script boilerplate.We want to install the two packages we downloaded into the image earlier.We configureto be launched by svscan and respond to DNS requests from the LAN.We could have created this user while creating the disk image, but since we needed to have a firstboot script anyway it was easier to do it here. Theoption means "read the password from standard input", which is why we're echoing it in from there.Part of the rc.d "firstboot" mechanism is to allow scripts to ask for the system to be rebooted after the first boot (and all the associated system initialization) is complete. In this case, we need to reboot in order to have svscan and isc-dhcpd running (since they weren't installed yet when the boot process started).More boilerplate. The rc.d script must be executable.The sentinel filetells FreeBSD that the system is booting for the first time and "firstboot" scripts should be run. At the end of the first boot,deletes this file.The EdgeRouter Lite boot loader expects to launch a Linux kernel which is found atwithin a FAT32 filesystem. Fortunately, it doesn't check that the kernel it's launching is Linux... so we create a FAT32 filesystem and drop a FreeBSD kernel in, named "/vmlinux.64" so that the EdgeRouter Lite boot loader launches it for us. (Our kernel is only about 10 MB, but the minimum size for a FAT32 filesystem is 33 MB.)We use thetool to create a UFS filesystem from the installed FreeBSD tree. The MIPS64 hardware is big-endian, and UFS is not endian-agnostic, so we need to tellto create a big-endian filesystem; this also means that (assuming we're using a little-endian system to build this disk image) we can't mount the filesystem on the system we're using to create it.The EdgeRouter Lite boot loader expects the kernel to be found on the first MBR slice; and the FreeBSD ERL kernel configuration expects the root filesystem to be found atso we'd better put it there.Once we finish building the disk image, we don't need our staging tree or the separate filesystem images any more.

Disqus