UPDATE 2019 / 2020

There are official ARM64 binaries of Ceph that you can run on a 64-bit version of Ubuntu 18.04

Important: I consider this page obsolete. I will keep it up for transparency's sake

Introduction

In this blog post I'll show you how to compile Ceph Luminous for the Raspberry Pi 3B+.

If you follow the instructions below you can compile Ceph on Raspbian. A note of warning: we will compile Ceph on the Raspberry Pi itself which takes a lot of time.

Ubuntu has packages for Ceph on armhf but I was never able to get Ubuntu working properly on the Raspberry Pi 3B+. Maybe that's just me and I did something wrong. Using existing Ceph packages on Ubuntu would probably be the fastest way to get up and running on the Raspberry Pi if it works for you.

This is my test Ceph cluster:

3 x Raspberry Pi 3B+ as Ceph monitors. 4 x HP Microserver as OSD nodes. 4 x 4 x 1 TB drives for storage (16 TB raw) 3 x 1 x 250 GB SSD (750 GB raw) 2 x 5-port Netgear switches for Ceph backend network (bonding)

For the impatient

If you just want the packages you can download this file and you'll get a set of .deb files which you need to install on your Raspberry Pi.

SECURITY WARNING: these packages are created by me, an unknown, untrusted person on the internet. As a general rule you should not download and install these packages as they could be malicious for all you know. If you want to be safe, compile Ceph yourself.

Skip to the section about installing the packages at the end for further installation instructions.

The problem with compiling Ceph for armhf

There are no armhf packages for Ceph because if you try to compile Ceph on armhf the compiler (gcc) will run out of virtual memory (about three gigabytes).

The solution

Daniel Glaser discovered that he could actually compile Ceph on armhf by using Clang/LLVM as the C++ compiler. This compiler seems to use less memory and thus stay within the 3 GB memory boundary. This is why he and I were able to compile Ceph.

How to compile Ceph for armhf - preparation

The challenge: one gigabyte of memory

The Raspberry Pi 3B+ has only one gigabyte of memory but we need more. The only way to add memory is to use swap on disk, as far as I know.

If you use storage as a substitute for RAM memory, you need fast storage, so it's really recommended to use an external SSD drive that you connect through USB. You also may need sufficient storage, I'd recommend 20+ GB.

SD memory cards are not up to the task regarding being used as swap. You'll wear them out prematurely and performance is abysmal. You should really use an external SSD.

Preparing the external SSD

Attach the SSD drive to the Raspberry Pi with USB

The SSD will probably show up as '/dev/sda'. mkfs.xfs /dev/sda -f ( this will erase all contents of the SSD ). mkdir /mnt/ssd mount /dev/sda /mnt/ssd

Creating and activating swap

cd /mnt/ssd dd if=/dev/zero of=swap.dd bs=1M count=5000 swapon /mnt/ssd/swap.dd swapoff /var/swap

By default, Raspbian configures a 100 MB swap file on /var/swap. In order to increase performance and protect the SD card from wearing out, please don't forget this last step to disable this swap file on the SD card.

Extra software

I would recommend installing 'htop' for real-time monitoring of cpu, memory and swap usage if you like to do so.

apt-get install htop

How to compile Ceph for armhf - building

Installing an alternative C++ compiler (Clang/LLVM)

As part of Daniel's instructions, you need to compile and install Clang/LLVM. I followed his instructions to the letter, I have not tested the Clang/LLVM packages made available through apt.

Compiling Clang/LLVM takes a lot of time. It took 8 hours to compile LLVM/Clang on a Raspberry Pi 3B+ with make -j3 to limit memory usage.

real 493m38.472s user 1223m39.063s sys 45m45.748s

I'll reproduce the steps from Daniel here:

apt update apt install - y build - essential ca - certificates vim git apt install libcunit1 - dev libcurl4 - openssl - dev python - bcrypt python - tox python - coverage cd / mnt / ssd mkdir git && cd git git clone https : // github . com / llvm - mirror / llvm . git cd llvm / tools git clone https : // github . com / llvm - mirror / clang . git git clone https : // github . com / llvm - mirror / lld . git cd / tmp mkdir llvm - build && cd llvm - build cmake - G "Unix Makefiles" - DCMAKE_BUILD_TYPE = Release - DLLVM_TARGETS_TO_BUILD = ARM / mnt / ssd / git / llvm / make - j3 make install update - alternatives --install /usr/bin/cc cc /usr/local/bin/clang 100 update - alternatives --install /usr/bin/c++ c++ /usr/local/bin/clang++ 100 update - alternatives --install /usr/bin/cpp cpp /usr/local/bin/clang-cpp 100

You may chose to build in some other directory, maybe on the SSD itself. I'm not sure if that makes a big difference. Be carefull when using /tmp as all contents are lost after a reboot.

Obtaining Ceph

There are two options: 1. clone my Luminous fork containing the branch 'ceph-on-arm' which incorporates all the 'fixed' files that make Ceph build with Clang/LLVM.

Clone the official Ceph repo and use the luminous branche. Next, you edit all the relevant files and make the changes yourself. Here you can find a list of all the files and the changes made to them.

I would recommend to just git clone ceph like this:

cd /mnt/ssd git clone https://github.com/louwrentius/ceph cd ceph git checkout ceph-on-arm git reset --hard git clean -dxf git submodule update --init --recursive

Now we first need to install a lot of dependancies on the Raspberry Pi before we can build Ceph.

run ./install-deps.sh

This will take some time as a ton of packages will be installed. Once this is done we are ready to compile Ceph itself.

Building Ceph

So to understand what you are getting into: it took me about 12 hours to compile Ceph on a Raspberry Pi 3 B+

real 717m31.457s user 1319m50.438s sys 58m7.549s

This is the command to run:

./make-debs.sh

If you want to monitor cpu and memory usage, you can use 'htop' to do so.

If for some reason the compile proces does fail and you may have to restart compiling ceph after you made some adjustments:

(you may have to adjust the folder name to match your ceph version)

cd /tmp/release/Raspbian/WORKDIR/ceph-12.2.9-39-gd51dfb14f4 < edit the relevant files here > dpkg-buildpackage -j3 -us -us -nc

Once this process is done, you will find a lot of .deb packages in your /tmp/release/Raspbian/WORKDIR folder.

Warning If you do use /tmp, the first thing to do is to copy all .deb files to a safe location because if you reboot your Pi, you loose 12 hours of work.

Assuming that you copied all .deb files to a folder like '/deb' you just created, this is how you install these packages:

dpkg --install *.deb apt-get install --fix-missing apt --fix-broken install

This is a bit ugly but it worked fine for me.

You can now just copy over all the .deb files to other Raspbery Pi's and install Ceph on them too.

Now you are done and you can run Ceph on a Raspberry Pi 3B+.

Ceph monitors may wear out the SD card

Important Running a Ceph monitor node on a Raspberry Pi is not ideal. The core issue is that the Ceph monitor process writes data every few seconds to files within /var/lib/ceph and this may wear out the SD card prematurely. The solution would be to use an external usb hard drive mounted through USB or a regular ssd which is way more resilient to writes than a regular SD card.