A long while ago I posted about trying to play around with Ceph for some storage shenanigans. If you’ve been following along you may notice that I never followed up on that post. The reason is that I abandoned that project because I just couldn’t be bother to figure out why everything seemed to be broken. Initially I was trying to setup a 1 node test cluster, but it seems like it’s less work and less stress to just suck up the hardware for the base config of 3 nodes. But anyways, I have returned to ceph with a vengeance and finally made good progress!

Out of necessity for an upcoming project with the IT club I have been revisiting Ceph over the last two weeks, and pulling my hair out the whole time. The hope is to set up a simple 3 node cluster for CephFS to use with OpenStack (who knows, maybe we’ll do block storage too) and since I’m the resident sheep, I get the fun task of testing this stuff out before we get our hands on the hardware. But after much suffering I finally got the test cluster into an active+clean state!

With any luck I’ve actually learned a few things this last trek through and any further installs will go a lot more smoothly. If there is only one thing I learned though, it’s that Ceph is finicky about some key details, and versions are VERY important.

I have also learned that the base distro you pick for your OS can make a lot of difference as well. The actual ceph-deploy steps are the same across each distro, but the repos can be a nightmare to try and work out (ahem, Ubuntu). I tried CentOS 7, Ubuntu 16.04, Ubuntu 17.04 (Dead xP), and Ubuntu 18.04. Out of all of them CentOS 7 ended up being the winner, which is nice because it has very quickly become my favorite flavor of linux.

The other really big detail that kicked me in the ass was version numbers. The usual route to get started with ceph (on CentOS at least) is to pick up the epel-release package and then graph ceph-deploy as well. The problem is that the default package that I got was an older version of ceph-deploy (1.5.9 I think?) which is fine if you’re trying to install Ceph Jewel or one of the older distros, but I wanted Ceph 12 to be current. The crappy part is that the older version of ceph-deploy can’t actually install Luminous (version 12) and you need version 2.0.0 of ceph-deploy to make things happy. I only found this out after running through most of the installation steps and finding that each node has ceph V.10 installed instead of V.12. Luckily adding the package info for the newer release of Ceph and updating got ceph-deploy up to version 2.0.0 and then from there it was really as easy as running the install command again only specifying the newer release of Ceph. I’m actually super surprised that the upgrade went that smoothly.

The last bit that really burned me were the disks that I wanted to use for the nodes. My test setup is super simple with 3 nodes, each with a 1TB disk dedicated just to ceph data. Each disk was in /dev/sdb and from what I understood the commands for creating an OSD were supposed to take care of everything on the disk, but when I tried to create the OSD I kept getting an error that /dev/sdb was being blocked by a filter. I’m still pretty sure that this is actually an issue with a filter somewhere, but I did find a workaround. By create a 1TB partition on each disk at /dev/sdb1 and passing that instead I was actually able to get all the OSDs running.

After getting all of that worked out I still had one little thing keeping me from achieving the healthy state. I’m pretty sure this is related to the fact that I actually installed Jewel at first instead of Luminous because the health warning was that the OSDs required a version that was less than Luminous. I was able to resolve this with a single command and now it reports as all healthy.

So now that the cluster is active+clean I have a few data pools set up to get CephFS running, and after just a little testing and tweaking I am able to mount the filesystem on another VM and write a test file. To double check I mounted the FS from each ceph node one at a time, and sure enough the file is there on each one! And with that it’s now 2:19AM and I have work in a little less than 7 hours.

I have to say that I’m still a little bitter at how difficult it was to get to this state, but I guess I did the same thing with OpenStack so it’s not really that bad. I had played with the idea of switching to glusterfs instead since it’s so much easier to install, but then I would have to mess with zfs or something to get what I want, and I don’t really want to open that can of worms right now. Being on the other end of things now I think Ceph is a really cool project and I like what it can do, but it seems to need just a little help on the documentation front.

Oh well, maybe I can do some more detailed write ups later on about what I actually had to do. I’ll have to tear this cluster down and rebuild it anyways just to be sure I can reproduce it, so we’ll see where I go with that.

But for now, it’s bed. Keep you head screwed on my friends!