From Sabayon Wiki

Why

Why all this? Yeah, good question. First of all, the world is changing and ARM devices are everyday more important. The second reason is that the same devices, have a great power consumption performance and are well suited for embedded, SOHO Linux servers and why not small HTPC.

The first goal of the Sabayon ARM experiment is to produce an image for the BeagleBone (hopefully working on other OMAP ARMv7a devices with little modification) that can transform that little device into a SOHO Linux server (NFS, Samba, Apache, Bittorrent are the main targets).

The second goal will be hopefully providing the whole X.Org stack and some audio/video apps.

Glossary

Host system

It is your own Linux system, the one with tons on GBs of RAM and lots of CPU power.

Target system

It is the actual ARM device.

Bootstrapping Gentoo on BeagleBones

Since that’s the only ARM board I have as of the time of writing this, the whole guide will be focused on OMAP devices produced by BeagleBoard.org, and in particular, on the BeagleBone.

Getting a Gentoo Stage3 tarball

Getting a Gentoo Stage3 tarball is easy (god bless Gentoo!), have a look here:

http://distfiles.gentoo.org/releases/arm/autobuilds/

Just pick the one for your architecture (and thus CHOST) and unpack it into /mnt/gentoo on your BeagleBone. In order to make sure to have enough space, you may consider to use an external SATA hard-drive or cheaper Class-10 SDHC memory (8GB suggested) (connected via stupid USB reader).

I strongly suggest to drop all the running BeagleBone services (node.js and others) and replace dropbear with OpenSSH, or all your “nohupped” processes will be killed once you logout from ssh.

Setting up a swap file

Those devices have limited RAM, so some swap might save your ass at times (besides causing trashing driving you crazy as well).

Let’s create a 512Mb swap file:

dd if=/dev/zero of=/mnt/swap.device bs=1024000 count=512 mkswap /mnt/swap.device swapon /mnt/swap.device

Compiling through qemu-user

Compiling on a real ARM device sucks due to RAM, disk and also redoundancy limitations. The solution is to use a qemu-user chroot on a build server. The first step is to copy (via rsync -H -A -X --numeric-ids) the whole chroot to the build server and install app-emulation/qemu-user-static from the sabayon overlay. At this point we need three things: a qemu-user wrapper executable (/usr/bin/qemu-static-arm-binfmt), /usr/bin/qemu-static-arm (qemu-user executable) copied inside the ARM chroot and registering the ARM ELF format against binfmt_misc filesystem on the kernel running on the same machine (the build server).

qemu-user wrapper executable

Install app-emulation/qemu-user-static and copy /usr/bin/qemu-static-arm-binfmt to /mnt/<your chroot>/usr/bin/ (make sure it's executable). The wrapper executable is ready!

The ARMEL interpreter for binfmt_misc

We need to tell the kernel how to interpret the ARM ELF executable binary format. qemu-user will be our on-the-fly translator. Just type this as root on the build server that’s hosting the chroot:

echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-static-arm-binfmt:P' > /proc/sys/fs/binfmt_misc/register

If it works, you will see:

# cat /proc/sys/fs/binfmt_misc/arm enabled interpreter /usr/bin/qemu-static-arm-binfmt flags: P offset 0 magic 7f454c4601010100000000000000000002002800 mask ffffffffffffff00fffffffffffffffffeffffff

qemu-static-arm executable

With app-emulation/qemu-user-static comes /usr/bin/qemu-static-arm that has to be copied inside the chroot (into /usr/bin/). This is the actual CPU instruction translator.

Chrooting

Now it’s time to enter the chroot, if everything went fine, it’s just a matter of:

chroot /mnt/chroot

And we’re done!

Of course, you can combine a qemu-user chroot with distcc and get a great boost. Remember that you’re emulating an ARM CPU inside the chroot.

Cross compiler on your host system

In order to be able to compile stuff via distcc, we need to setup a cross-compiler on the host system. You need to install crossdev package in order to achieve that, as explained on the Gentoo Linux Embedded Handbook.

Make sure to match the toolchain versions (gcc, binutils, glibc, kernel headers) of the target system, or the cross compiler won’t work.

You can create the cross compiler for CHOST=armv7a-hardfloat-linux-gnueabi as follows:

crossdev -t armv7a-hardfloat-linux-gnueabi [--binutils <some ver>] [--gcc <some ver>]

Once created the cross compiler with crossdev, you can test if it works by compiling the Linux kernel as follows (in this case we have an armv7a cross compiler):

make ARCH=arm defconfig make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- uImage

If it doesn’t complain, then we have the proof that’s all good!

Setting up distcc-pump

Of course, the target system and the host system need to communicate each others.

That is the only way to let the host system compile stuff for the target system.

Having said that, you just need to install distcc and configure /etc/conf.d/distccd as follows:

DISTCCD_OPTS="${DISTCCD_OPTS} --allow <your network + subnet> --listen <listen ip>" DISTCCD_OPTS="${DISTCCD_OPTS -N 0"

Start it up and you now have a working distcc daemon in place.

Cross-compile over distcc with emerge

Following this Gentoo Linux Cross Compiling Distcc Guide, I’ve been able to prepare some ready-to-use scripts inside the build.git repository.

One last thing you need to do on the target system by the way, is actually telling it (distcc) what are the available distcc peers and enable PUMP mode for them. You can do this by editing /etc/distcc/hosts as follows (see distcc man pages for more ways to declare hosts or networks):

192.168.1.3,cpp,lzo

In this case we just tell distcc on the target system to use the server at 192.168.1.3 (our host system IP address).

DistCC via SSH

DistCC via SSH can be considered safer (in terms of security) and perhaps a bit more reliable when used inside qemu-user. There is a good guide at [1]. To keep it simple, on any host system (compilation node), make sure to setup ssh access coming from the target device as root user (the one used by Portage for compilation) and then setup the following make.conf vairables:

# distcc configuration DISTCC_SSH="/etc/distcc/distcc-ssh" # <host>/<number of jxobs> DISTCC_HOSTS="localhost/2 [email protected]/14,lzo,cpp"

There is no need to have the distccd daemon listening.

/etc/distcc/distcc-ssh is like this:

[1] http://en.gentoo-wiki.com/wiki/Distcc_over_SSH

If you have troubles with DistCC and SSH regarding TCP_CORK, try disabling setting DISTCC_TCP_CORK=0 in the host system distcc user environment.

armv7a-emerge

This is an emerge wrapper that properly configures /usr/lib/distcc/bin in order to make it work with build systems hardcoding CC (and others) variables.

Basically, it sets up proper Portage FEATURES, makes sure that distcc-pump is fine and spawns emerge on your behalf.

You can use it the same way you use emerge itself.

You can find it here: http://git.sabayon.org/build.git/tree/bin/armv7a-emerge

Compiling the Kernel

As also written before, this guide focuses on BeagleBoard and BeagleBone ARMv7 OMAP devices. So, we’re going to build the kernel, using genkernel and the ebuilds on the sabayon-distro overlay for this specific architecture.

The same kernel git repo is available on git.sabayon.org and contains patches for the Beagle* devices not yet (3.1 kernel) merged into mainline kernel.

Sources: sys-kernel/beaglebone-sources Kernel: sys-kernel/linux-beaglebone

Let’s dance

/sabayon/bin/armv7-emerge -av sys-kernel/beaglebone-sources /sabayon/bin/armv7-emerge -av sys-kernel/linux-beaglebone

This will require genkernel and lzop (they’ll be installed as dependency).

Cross-compiler via DistCC under qemu-user

How about using a compiler that is not running inside qemu-user? This can be tricky, since distcc via ssh might give you NOTSOCKET COMPILE_ERROR, and disabling TCP_CORK won't fix it. If you are into all this mess, just go back to the old dear distccd and use "127.0.0.1" to point to localhost daemon. Using "localhost" won't work, since for distcc it means "no distributed compilation".

Partitioning a MMC

Let’s say that we have a working chroot and kernel and we want to deploy it on a MMC. The advice here is to buy Class 10 SDHCs and an USB reader for them.

Any size is fine, we will use mkcard.txt from molecules.git repository.

This will create a partition layout composed by two partitions: the vfat partition used for bootstrapping the device (sdX1 or mmcblkXp1) and the root partition (sdX2 or mmcblkXp2). You could do the whole process by hand... but that’s another matter.

Make sure to have sys-devel/bc installed, otherwise mkcard.txt will fail.

So, just insert the empty MMC into your USB reader, find out the device name (let’s say /dev/sde here) and type as root:

sh mkcard.txt /dev/sde

If all goes well, the two partitions will be created and your MMC is ready to host Sabayon.

Booting on ARM (BeagleBone)

Booting on ARM sucks. But GRUB2 sucks more IMHO.

These ARM boards use a special vfat partition in order to be able to bootstrap, there are 4 important things in this context:

The MLO image (contains the x-loader code that starts u-boot -- like grub stage1 crap). The boot parameters file (uEnv.txt file) The u-boot image file (u-boot.img) The kernel image (uImage file)

MLO (Mmc LOader)

You could compile your own MLO, but why? Usually, BeagleBoard people at least, provide their own MLO and there isn’t much need to cook your own off x-loader crap.

So, just grab it from molecules.git repo here and place inside the vfat partition (the boot partition).

If you are interested in building it, have a look at http://gitorious.org/x-load-omap3 . But anyway, the u-boot repo listed in the u-boot.img chapter contains the buildsystem for MLO (it actually builds the MLO).

uEnv.txt

In this configuration, we want to load the kernel image (uImage) directly from the root filesystem. This way we can control the kernel version without touching the boot partition any further.

For the BeagleBone, the uEnv.txt file is quite simple and you can find it here.

Just copy it over to the vfat partition (the boot partition).

u-boot.img

The same about MLO applies to u-boot.img, but this is easier to get working. If your vendor doesn’t provide a read-to-use u-boot.img, you can make your own (making sure that the git repo you are going to clone has support for your hardware).

If you go ahead reading this, take into account that you need a cross compiler for your ARM arch.

In our case, with the BeagleBone, it’s just a matter of using the official u-boot git repo at git://git.denx.de/u-boot.git:

git clone git://git.denx.de/u-boot.git make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- distclean make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- am335x_evm_config make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi-

This will generate a final u-boot.img file that you need to copy to the vfat partition (the boot partition).

You may need the following patch in order to boot Sabayon ARM images (expecting /boot/uImage and /boot/uInitrd for kernel and ramdisk files on the rootfs):

diff --git a/include/configs/am335x_evm.h b/include/configs/am335x_evm.h index 0170e11..89a46eb 100755 --- a/include/configs/am335x_evm.h +++ b/include/configs/am335x_evm.h @@ -46,7 +46,9 @@ #define CONFIG_EXTRA_ENV_SETTINGS \ "bootfile=uImage\0" \ + "ramdiskfile=uInitrd\0" \ "loadaddr=0x82000000\0" \ + "rdaddr=0x81000000\0" \ "script_addr=0x81900000\0" \ "console=ttyO0,115200n8\0" \ "mmc_dev=0\0" \ @@ -75,6 +77,8 @@ "importbootenv=echo Importing environment from mmc ...; " \ "env import -t $loadaddr $filesize\0" \ "mmc_load_uimage=fatload mmc ${mmc_dev} 0x80007fc0 ${bootfile}\0" \ + "mmc_load_ext_uimage=ext2load mmc ${mmc_dev}:2 ${loadaddr} /boot/${bootfile}\0" \ + "mmc_load_ext_initrd=ext2load mmc ${mmc_dev}:2 ${rdaddr} /boot/${ramdiskfile}\0" \ "optargs=\0" \ "bootargs_defaults=setenv bootargs " \ "console=${console} " \ @@ -101,8 +105,14 @@ "nfsroot=${serverip}:${rootpath},${nfsopts} rw " \ "ip=dhcp\0" \ "mmc_boot=run mmc_args; " \ - "run mmc_load_uimage; " \ - "bootm\0" \ + "if run mmc_load_uimage; then " \ + "bootm;" \ + "fi;" \ + "if run mmc_load_ext_uimage; then " \ + "run mmc_load_ext_initrd;" \ + "bootm;" \ + "fi;" \ + "\0" \ "nand_boot=echo Booting from nand ...; " \ "run nand_args; " \ "nand read.i ${loadaddr} ${nand_src_addr} ${nand_img_siz}; " \ @@ -138,6 +148,14 @@ "run mmc_args;" \ "bootm;" \ "fi;" \ + "if run mmc_load_ext_uimage; then " \ + "run mmc_args;" \ + "if run mmc_load_ext_initrd; then " \ + "bootm ${loadaddr} ${rdaddr};" \ + "else " \ + "bootm ${loadaddr};" \ + "fi;" \ + "fi;" \ "fi;" \ "run nand_boot;" \

uImage

Kernel images coming from sabayon-kernel.eclass install an uImage file directly.

An uImage is generated from a zImage using mkimage (available in dev-embedded/u-boot-tools).

Example:

mkimage -A arm -O linux -T kernel -C none -a <load address> -e <kernel entry point address> -n Linux -d arch/arm/boot/zImage uImage

But there is no real need, because kernel target “uImage” already takes care of it, if mkimage is installed.

The uImage file, given the current boot files layout, can be placed inside the root filesystem. This has the big advantage of letting the package manager handle kernel updates properly without carying about about partition crapshit.

In order to manage the actual /boot/uImage file (on the root filesystem), I wrote app-admin/eselect-uboot, that makes possible to switch between available kernel binaries. It works by handling /boot/uImage as a symlink, that’s it.

Once all this is in place, we’re ready to boot! (almost).

Serial console

As explained in the Tips and tricks chapter, the first source of info when something goes wrong at boot is the serial console.

But, in order to make it work and allow login, you need to tweak /etc/inittab.

For the BeagleBone the following line should be edited (changing the current s0:... line):

s0:12345:respawn:/sbin/agetty 115200 ttyO0 vt100



And of course, add ttyO0 to /etc/securetty.

Root password

That’s the stupidest thing but sure enough, you will forget to set a root password.

Networking

Make sure to setup the net.eth0 symlink and perhaps add net.eth0 to the default runlevel, as well as sshd if you want it.

Language

You can set your LC_*, LANG and LANGUAGE variables in /etc/env.d/02locale file.

/etc/fstab

Fortunately, fstab is something belonging to the past, but still there is the need to have it configured in a minimal way. Here is a good and reliable version of it:

# <fs> <mount> <type> <opts> <dump/pass> rootfs / rootfs rw 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0

Tips and tricks

Here are some tips and tricks that could save your ass (and time).

How to connect to the BeagleBone via mini-USB

modprobe ftdi_sio vendor=0x0403 product=0xa6d0 screen /dev/ttyUSB1 115200 # (or USB0)

How to test-compile a kernel on x86 for ARM

A cross compiler must be available on the system (see the Crossdev section above).