So I’ve been thinking about this since 2017, but only yesterday signed up for an account and played around with the ARM64 offering.

Turns out it’s pretty great! KVM boots into UEFI, there’s a local VirtIO disk attached, no NBD junk required. So we can definitely run FreeBSD.

I managed to “depenguinate” a running instance, the notes are below.

Would be great if Scaleway offered an official image instead

For some reason, unlike on x86, mounting additional volumes is not allowed on ARM64 instances. So we’ll have to move the running Linux to a ramdisk using pivot_root and then we can do whatever to our one and only disk.

Spin up an instance with Ubuntu Zesty and ssh in.

Prepare the system and change the root to a tmpfs:

apt install gdisk mount -t tmpfs tmpfs /tmp cp -r /bin /sbin /etc /dev /root /home /lib /run /usr /var /tmp mkdir /tmp/proc /tmp/sys /tmp/oldroot mount /dev/vda /tmp/oldroot mount --make-rprivate / pivot_root /tmp /tmp/oldroot for i in dev proc sys run; do mount --move /oldroot/$i /$i; done systemctl daemon-reload systemctl restart sshd

Now reconnect to ssh from a second terminal (note: rm the connection file if you use ControlPersist in ssh config), then exit the old session. Kill the old sshd process, restart or stop the rest of the stuff using the old disk:

pkill -f notty sed -ibak 's/RefuseManualStart.*$//g' /lib/systemd/system/dbus.service systemctl daemon-reload systemctl restart dbus systemctl daemon-reexec systemctl stop user@0 ntp cron systemd-logind systemctl restart systemd-journald systemd-udevd pkill agetty pkill rsyslogd

Check that nothing is touching /oldroot:

lsof | grep oldroot

There will probably be an old dbus-daemon, kill it.

And finally, unmount the old root and overwrite the hard disk with a memstick image:

umount -R /oldroot wget https://download.freebsd.org/ftp/snapshots/arm64/aarch64/ISO-IMAGES/12.0/FreeBSD-12.0-CURRENT-arm64-aarch64-20180719-r336479-mini-memstick.img.xz xzcat FreeBSD-12.0-CURRENT-arm64-aarch64-20180719-r336479-mini-memstick.img.xz | dd if=/dev/stdin of=/dev/vda bs=1M

(Look for the newest snapshot here, don’t copy paste the July 19 link above if you’re reading this in the future. Actually maybe use a release instead of CURRENT…)

Now, fix the GPT: move the secondary table to the end of the disk and resize the table.

It’s important to resize here, as FreeBSD does not do that and silently creates partitions that won’t persist across reboots

gdisk /dev/vda x e s 4 w y

And reboot . (You might actually want to hard reboot here: for some reason on the first reboot from Linux, pressing the any-key to enter the prompt in the loader hangs the console for me.)

I didn’t have to go into the ESC menu and choose the local disk in the boot manager, it seems to boot from disk automatically.

Now we’re in the FreeBSD EFI loader.

For some reason, the (recently fixed?) serial autodetection from EFI is not working correctly. Or something.

So you don’t get console output by default.

To fix, you have to run these commands in the boot loader command prompt:

set console=comconsole,efi boot

(Ignore the warning about comconsole not being a valid console.

Since there’s at least one ( efi ) that the loader thinks is valid, it sets the whole variable.)

(UPD: shouldn’t be necessary in the next snapshot)

Now it’s a regular installation process!

When asked about partitioning, choose Shell, and manually add a partition and set up a root filesystem:

gpart add -t freebsd-zfs -a 4k -l zroot vtbd0 zpool create -R /mnt -O mountpoint=none -O atime=off zroot /dev/gpt/zroot zfs create -o canmount=off -o mountpoint=none zroot/ROOT zfs create -o mountpoint=/ zroot/ROOT/default zfs create -o mountpoint=/usr zroot/ROOT/default/usr zfs create -o mountpoint=/var zroot/ROOT/default/var zfs create -o mountpoint=/var/log zroot/ROOT/default/var/log zfs create -o mountpoint=/usr/home zroot/home zpool set bootfs=zroot/ROOT/default zroot exit

(In this example, I set up ZFS with a beadm -compatible layout which allows me to use Boot Environments.)

In the post-install chroot shell, fix some configs like so:

echo 'opensolaris_load="YES"' >> /boot/loader.conf echo 'console="comconsole,efi"' >> /boot/loader.conf echo 'vfs.zfs.arc_max="512M"' >> /boot/loader.conf sysrc zfs_enable=YES exit

(Yeah, for some reason, the loader does not load zfs.ko 's dependency opensolaris.ko automatically here. idk what even. It does on my desktop and laptop.)

Now you can reboot into the installed system!!

Here’s how you can set up IPv6 (and root’s ssh key) auto configuration on boot:

pkg pkg install curl curl https://raw.githubusercontent.com/scaleway/image-tools/master/bases/overlay-common/usr/local/bin/scw-metadata > /usr/local/bin/scw-metadata chmod +x /usr/local/bin/scw-metadata echo '#\!/bin/sh' > /etc/rc.local echo 'PATH=/usr/local/bin:$PATH' >> /etc/rc.local echo 'eval $(scw-metadata)' >> /etc/rc.local echo 'echo $SSH_PUBLIC_KEYS_0_KEY > /root/.ssh/authorized_keys' >> /etc/rc.local echo 'chmod 0400 /root/.ssh/authorized_keys' >> /etc/rc.local echo 'ifconfig vtnet0 inet6 $IPV6_ADDRESS/$IPV6_NETMASK' >> /etc/rc.local echo 'route -6 add default $IPV6_GATEWAY' >> /etc/rc.local mkdir /run mkdir /root/.ssh sh /etc/rc.local

One remaining issue: incoming TCP connections (like ssh, or simply testing with nc ) from the public IPv4 address don’t work — I see incoming packets in tcpdump but no replies o_0 (IPv6 works perfectly, at least)

And to fix incoming TCP connections, configure the DHCP client to change the broadcast address:

echo 'interface "vtnet0" { supersede broadcast-address 255.255.255.255; }' >> /etc/dhclient.conf killall dhclient dhclient vtnet0

Other random notes:

keep in mind that -CURRENT snapshots come with a debugging kernel by default, which limits syscall performance by a lot, you might want to build your own with config GENERIC-NODEBUG

also disable heavy malloc debugging features by running ln -s 'abort:false,junk:false' /etc/malloc.conf (yes that’s storing config in a symlink)

(yes that’s storing config in a symlink) you can reuse the installer’s partition for swap