Bodhi mash

This is mostly Fedora release engineering work specific post. Feel free to skip if you are not into Fedora land.

I started packaging for Fedora from 2006, back in Fedora Extras repository days. Now, we have only one big repo. Packagers can just submit their new packages or updates to this repository. Now, to do so, first, we will have to get the package reviewed (in the case of new packages). Later, after getting a git repo for the spec file, one can build the package in Koji, this will create the RPMs and the SRPM in the koji server. Next, we mark the package for a push to the testing repository (or after certain criteria, mark them to push to the stable repository). For rawhide (the latest of everything) repository, we don't have to do this, everything we build for rawhide will be automatically pushed.

What happens after we mark the package ready for push?

Every day, someone from the Fedora release engineering team is on push duty. This person calls bodhi-push commands, which in turn generates a fedmsg, which then consumed by a the consumer, defined in bodhi masher.py to generate new repos from the updates, and also mark the packages properly in koji. It also composes the ostree tree. We will now dig into the source of this command to see what all happens.

The work starts at line 271 At line 290, it initiates the state. There are use cases when it resumes from the middle of a previous mash, the if-else block at line 301 helps to load or save the state of the mash accordingly. To load or save it actually reads up a JSON file and a few other steps.

Next, at line 306, it calls a function load_updates. This function talks to the database and finds out the list of packages which need to be pushed (the final repo will be generated from all the packages in that particular tag) for updates. Now all of this work happens based on a unique tag, like f25-updates. The verify_updates function makes sure that list of the updates has the right updates for the current tag (it can happen that someone tries to mark an F24 build as the F25 update).

We have to do some more checks if this push is for stable repositories, like if the build has enough positive karma, or has it spent 7 days in the testing repository. At line 310, the perform_gating function does these gating checks.

At line 312, in the determine_and_perform_tag_actions functions, we talk to Koji and update the tags in Koji as required. Either it adds a new tag or moves the build to a new tag. Like, from candidate to the testing. After that we update the bugzilla bug entries if there are any update with a bug associated, and in the next few lines we remove any extra tags, and also update the comps files (from the git repo). At line 321, we create a new thread which does the real mash work using the mash command from the package with the same name. While we wait for this thread to finish on line 330, we have already created some digest mail information, and other updateinfo (in the uinfo variable, which is the content of updateinfo.xml). In line 337 we check if we are supposed to do build ostree repos for atomic, if yes, then we call the compose_atomic_trees function.

Then we sync the newly generated repo with the master mirror, and wait for it to finish (on line 345). After that it sends out fedmsg notifications, modifies the bugs in the bugzilla, adds comments to the koji updates, and also does the announcement emails (the function calls are documented).

The main reason I wanted to write this post is the upcoming change where we will stop using the mash command, and instead call pungi to do the work. As pungi can also do ostree tree compose.

I want to specially thank Patrick who helped me to understand this piece of code (as he does for many other things).