I have held out on building packages on FreeBSD for a long time.

My experience with portmaster and portupgrade was not perfect, but it was pretty consistent, and I always knew how to recover from any failures. I normally had systems build packages via portmaster during the early am, or through some automated process that I didn’t have to look at.

At this point though, pkgng has become a much better tool than the pkg_ tools, and while portmaster and portupgrade support registering packages with pkgng, I felt this was a good time to start building local packages for our infrastructure.

The main tools I use are:

Poudriere : Sort of like Tinderbox, but it is specifically for pkgng (I didn’t like tinderbox and pkgng integration either)

Jenkins CI : I use jenkins to kick off and manage building packages, syncing them to a web server, and re-generating the package index

EJBCA : Our internal CA that I setup. I used it to generate the OpenSSL keypair that is used to sign packages. Its a wonderful tool and I like to mention it as much as possible. Its too complex to discuss here, perhaps at another date

Lets start with Poudiere

Poudriere

ZFS Setup

You’ll need a ZFS pool, which means you need zfs enabled:

kldload zfs echo zfs_enable = YES > /etc/rc.conf.d

I do all of this on a VM (two in fact, I build desktop packages separate from server packages), and each vm has a dedicated virtual disk for poudriere buils:

zpool create data ada1

Install

Install ports-mgmt/poudriere through whatever mechanisms you want to (but I bet after this, you’ll start using pkg :) )

portinstall ports-mgmt/poudriere

cd /usr/ports/ports-mgmt/poudriere

pkg install poudriere

Configuration

A .sample file will get installed, and that has a few more options, but we’ll keep it basic:

ZPOOL = data FTPHOST = ftp4.us.freebsd.org RESOLV_CONF = /etc/resolv.conf POUDRIERE_DATA = /data/poudriere_data BASEFS = /data/poudriere USE_PORTLINT = no USE_TMPFS = yes DISTFILES_CACHE = /usr/ports/distfiles CRONDIR = ${ BASEFS } /cron SVN_HOST = svn0.us-west.FreeBSD.org NO_PACKAGE_BUILDING = no PKG_REPO_SIGNING_KEY = /usr/local/etc/poudriere.d/pkgng.key

The ZPOOL value should be the name of the zfs pool created above. I like to use the name data, its just a preference of mine. The PKG_REPO_SIGNING_KEY is your key, and I just placed it in the .d directory.

Also, you may want to pick an ftp and svn host that is geographically close to you, but thats optional.

Once that is done, you’re all ready to create jails and a ports tree

Jails

Poudriere needs to build packages from a FreeBSD jail, and this is really cool. You can have a 9.1, 8.2, 10 (aka, CURRENT) and mix between i386 and amd64 platforms.

I myself only build 9.1-RELEASE amd64 packages. Our older 9.x systems are still using Puppet and Portmaster. When I upgrade them, I’ll switch them to Salt and pkgng.

First, create a ports tree for your jail(s) to use:

$ poudriere ports -c $ poudriere ports -l PORTSTREE METHOD PATH default portsnap /data/poudriere/ports/default

The default update method is to use portsnap, but you can also use svn:

$ poudriere ports -c -m svn

Next, we create a jail. I’ll create a 9.1-RELEASE AMD64 jails:

$ poudriere jail -c -j 91amd64 -a amd64 -v 9.1-RELEASE $ poudriere jail -l JAILNAME VERSION ARCH METHOD SUCCESS FAILED IGNORED SKIPPED QUEUED STATUS 91amd64 9.1-RELEASE amd64 ftp 0 0 0 0 0 idle:

Building a Port

The command you’ll be working with is poudriere bulk. I use the -f flag to pass it a file, and I’ll show that later, but to test everything out go ahead and build a single port:

poudriere bulk -j 91amd64 sysutils/sysrc

Here is the output:

==== >> Mounting system devices for 91amd64 /etc/resolv.conf -> /data/poudriere/jails/91amd64/etc/resolv.conf ==== >> Starting jail 91amd64 ==== >> Mounting ports from: /data/poudriere/ports/default ==== >> Mounting packages from: /data/poudriere_data/packages/91amd64-default ==== >> Mounting /var/db/ports from: /usr/local/etc/poudriere.d/91amd64-options ==== >> Appending to /etc/make.conf: /usr/local/etc/poudriere.d/91amd64-make.conf ==== >> Populating LOCALBASE ==== >> Calculating ports order and dependencies ==== >> Sanity checking the repository ==== >> Deleting stale symlinks ==== >> Cleaning the build queue ==== >> Building 1 packages using 1 builders ==== >> [ 01 ] Starting build of sysutils/sysrc ==== >> [ 01 ] Finished build of sysutils/sysrc: Success ==== >> Stopping 1 builders ==== >> Creating pkgng repository tar: Removing leading '/' from member names Generating repo.sqlite in /usr/ports/packages/: done! ==== >> Cleaning up ==== >> Umounting file systems ==== >> Built ports: sysutils/sysrc ==== >> [ 91amd64 ] 1 packages built, 0 failures, 0 ignored, 0 skipped

We can now see that package in /data/poudriere_data/packages/91amd64-default/sysutils/sysrc-5.1.txz, and if you want to install if manually, just use pkg add:

$ pkg info -l -F /data/poudriere_data/packages/91amd64-default/sysutils/sysrc-5.1.txz sysrc-5.1 owns the following files: /usr/local/libexec/sysrc/include/messages.subr /usr/local/man/man8/sysrc.8.gz /usr/local/sbin/sysrc /usr/local/share/licenses/sysrc-5.1/BSD /usr/local/share/licenses/sysrc-5.1/LICENSE /usr/local/share/licenses/sysrc-5.1/catalog.mk /usr/local/share/sysrc/common.subr /usr/local/share/sysrc/sysrc.subr $ pkg add /data/poudriere_data/packages/91amd64-default/sysutils/sysrc-5.1.txz Installing sysrc-5.1... done

Otherwise, you can point a system to the generated repo file and use pkg install

Build-time options

Before we get into the entire automation process, did you notice this line at the begining of the build?

==== >> Mounting /var/db/ports from: /usr/local/etc/poudriere.d/91amd64-options ==== >> Appending to /etc/make.conf: /usr/local/etc/poudriere.d/91amd64-make.conf

Poudriere will first look in /usr/local/etc/poudriere.d/${JAIL_NAME}-options for the build time options for a port (make options).

For the most part, I take the default options except for a few very important builds like Samba, Nginx, Apache, etc…

To set those build time options, just run poudriere options -j 91amd64 net/samba36

The other thing poudriere does is look at your local /etc/make.conf and appends it to /usr/local/etc/poudriere.d/${JAIL_NAME}-make.conf

My make.conf looks like this:

WITH_PKGNG = yes WITHOUT_X11 = yes .if empty ( .CURDIR:M/usr/ports/databases/mongodb* ) CC = gcc CXX = g++ CPP = cpp .endif CC = clang CXX = clang++ CPP = clang-cpp

And, my make.conf for the vm that builds desktop pacakges looks like this:

WITH_PKGNG = yes CC = clang CXX = clang++ CPP = clang-cpp WITHOUT_NOUVEAU = yes WITH_NEW_XORG = yes

As you can see, I make our server packages without X11 support, but the desktop packages get X11.

I try to build with clang, but there are a few cases when it just doesn’t work so I’ve attempted to use a little if statement (I’m not sure if it works thought…)

Managing with Salt

I have a fairly simple salt state to help the build process, and I only recently discovered that salt has a poudriere module to help manage everything I just wrote above.

Pillar Data

{% gist 5348213 %}

State

{% gist 5348166 %}

ZPOOL = data FTPHOST = ftp4.us.freebsd.org RESOLV_CONF = /etc/resolv.conf POUDRIERE_DATA = /data/poudriere_data BASEFS = /data/poudriere USE_PORTLINT = no USE_TMPFS = yes DISTFILES_CACHE = /usr/ports/distfiles CRONDIR = ${ BASEFS } /cron SVN_HOST = svn0.us-west.FreeBSD.org NO_PACKAGE_BUILDING = no PKG_REPO_SIGNING_KEY = /usr/local/etc/poudriere.d/pkgng.key

converters/p5-JSON-PP databases/couchdb databases/rubygem-couchrest databases/mongodb databases/memcached databases/mysql55-client databases/mysql55-server databases/pecl-memcache databases/postgresql92-client databases/postgresql92-server ,,,

accessibility/atk accessibility/atkmm archivers/gtar archivers/p7zip archivers/rpm archivers/rpm2cpio archivers/squeeze archivers/unzip archivers/zip audio/alsa-lib ...

Jenkins

Setup

I based most of what I have off of this great three part article Continuous package building with poudriere and Jenkins under FreeBSD

I eventually re-worked it and did it my own way, which I just found simpler to grok and manage.

You’ll need to enable two Jenkin’s plugins:

Jenkins PostBuildScript Plugin

Jenkins Publish Over SSH Plugin

Once those are installed, you need to setup a ssh host under Jenkins -> Manage Jenkins -> Configure System -> Publish Over SSH

PkgNg Job

Create a new Jenkins job. I titled mine FreeBSD-9.1-amd64-server, and the job type is Build a free-style software project

The job has 5 simple steps:

Update the ports tree poudriere ports -u

Build the packages from the /root/plist file poudriere bulk -J 8 -j 91amd64 -f /root/plist

Show the jail information poudriere jails -i -j 91amd64

Rsync packages to separate web server: rsync -ave “ssh -i /home/jenkins-ci/.ssh/id_rsa” –force /data/poudriere_data/packages/91amd64-default/ jenkins-ci@pkgng:/data/www/pkgng/data/9.1-freebsd-amd64/server/

Rebuild the repo index pkg repo /data/www/pkgng/data/9.1-freebsd-amd64/server /data/www/pkgng/pkgng.key



To set that up in Jenkins, add these Build Steps:

Build Send files or execute commands over SSH SSH Publisher name: pkg-server Credentials: Username: jenkins-ci Passphrase/Password: ************* Path to key: /home/jenkins-ci/id_rsa.publish Transfers: Source files: /bin/sh Remote Prefix: Remote Directory: Exec command: sudo poudriere ports -u sudo poudriere bulk -J 8 -j 91amd64 -f /root/plist sudo poudriere jails -i -j 91amd64 Exec timeout: 0 (ms)

Post-build actions [PostBuildScript] - Execute a set of scripts Build steps SSH Publisher name: pkg-server Credentials: Username: jenkins-ci Passphrase/Password: ************* Path to key: /home/jenkins-ci/id_rsa.publish Transfers: Source files: /bin/sh Remote Prefix: Remote Directory: /root Exec command: /root/pkg-rsync.sh Exec timeout: 0 (ms) SSH Publisher name: pkgng Transfers: Source files: /bin/sh Remote Prefix: Remote Directory: /data/www/pkgng/data/9.1-freebsd-amd64/server/ Exec command: pkg repo . /data/www/pkgng/pkgng.key





Clients

Now that the packages are built, and then rsync’d over to a web server (I’m using nginx), your clients will need to have a usable pkg.conf, as well as the public key for the signed packages

Salt

In our basepkgs state, we manage the pkg.conf file, and push out the public certificate:

{% gist 5348798 %}

{% gist 5348827 %}

The pillar data for host.role is used to differentiate if a client will use the Desktop package repo (built with X11 support and CUPS), or the Server repo (trimmed down options and almost no support for X11, GTK+, CUPS, etc…)

The final file that ends up on the system will look like this:

PUBKEY: /usr/local/etc/pkg/ssl/pkgng.pub PACKAGESITE: https://pkgng.bayphoto.com/9.1-freebsd-amd64/server/

Now, we can finally update our repo index, and upgrade packages:

root@pkg-server:/root # pkg update Updating repository catalogue repo.txz 100% 275KB 275.2KB/s 275.2KB/s 00:01 root@pkg-server:/root # pkg upgrade Updating repository catalogue Repository catalogue is up-to-date, no need to fetch fresh copy New version of pkg detected; it needs to be installed first. After this upgrade it is recommended that you do a full upgrade using: 'pkg upgrade' The following packages will be upgraded: Upgrading pkg: 1.0.9_2 -> 1.0.11 The installation will require 6 kB more space 1 MB to be downloaded Proceed with upgrading packages [ y/N ] : y pkg-1.0.11.txz 100% 1524KB 1.5MB/s 1.5MB/s 00:01 Checking integrity... done Upgrading pkg from 1.0.9_2 to 1.0.11... done root@pkg-server:/root # pkg upgrade Updating repository catalogue Repository catalogue is up-to-date, no need to fetch fresh copy The following packages will be upgraded: Upgrading gamin: 0.1.10_4 -> 0.1.10_5 Reinstalling cairo-1.10.2_5,2 Reinstalling pango-1.30.1 Upgrading libxml2: 2.7.8_5 -> 2.8.0_1 Upgrading vim: 7.3.669 -> 7.3.669_1 Upgrading subversion: 1.7.8 -> 1.7.9 Upgrading gdk-pixbuf2: 2.26.5 -> 2.26.5_3 Upgrading portmaster: 3.14_9 -> 3.16 The installation will free 5 MB 12 MB to be downloaded Proceed with upgrading packages [ y/N ] : y gamin-0.1.10_5.txz 100% 70KB 69.6KB/s 69.6KB/s 00:00 cairo-1.10.2_5,2.txz 100% 524KB 523.8KB/s 523.8KB/s 00:00 pango-1.30.1.txz 100% 553KB 553.0KB/s 553.0KB/s 00:01 libxml2-2.8.0_1.txz 100% 774KB 774.1KB/s 774.1KB/s 00:00 vim-7.3.669_1.txz 100% 5253KB 5.1MB/s 2.3MB/s 00:01 subversion-1.7.9.txz 100% 4485KB 4.4MB/s 4.4MB/s 00:01 gdk-pixbuf2-2.26.5_3.txz 100% 518KB 518.0KB/s 518.0KB/s 00:00 portmaster-3.16.txz 100% 43KB 43.0KB/s 43.0KB/s 00:00 Checking integrity... done Upgrading gamin from 0.1.10_4 to 0.1.10_5... done Reinstalling cairo-1.10.2_5,2 done Reinstalling pango-1.30.1 done Upgrading libxml2 from 2.7.8_5 to 2.8.0_1... done Upgrading vim from 7.3.669 to 7.3.669_1... done Upgrading subversion from 1.7.8 to 1.7.9... done Upgrading gdk-pixbuf2 from 2.26.5 to 2.26.5_3... done Upgrading portmaster from 3.14_9 to 3.16... done root@pkg-server:/root #

Ta Da!

As a FreeBSD and general SysAdmin, this process has been pretty cool. I’ve discovered a few hiccups, so hopefully with the above information, you will not have to deal with those.