Fedora kernel git tree [Oct. 1st, 2014|06:48 pm] jwboyer

As mentioned in my last post, a number of people over the years have asked about an exploded source git tree for the Fedora kernel. We've never done this, primarily because our build tool wouldn't be able to use it, and it didn't really have any perceived value for the maintainers. However, it's come up enough times that I thought I'd give it a shot. Creating it was easy enough, but whether it winds up being useful to anyone is something that you'll need to help with. This post will cover what it is, how it works, where it is, and possible future plans.



For the TLDR, see the bottom.



What is it: This is maybe the git tree you are looking for

If you've ever worked on packaging software with RPM (or apt or a number of other package managers), there are two basic ways to do it. You can use the upstream release tarballs and apply patches via a spec file, or you can use a snapshot of the upstream development repository as the "tarball". The former is how the Fedora kernel is packaged, and it presents a nice separation of upstream and the patches Fedora has added. However, that means that every time a new upstream kernel is released we wind up rebasing our patch set on top of it. When doing the initial creation of the git tree, I thought about a number of ways to handle this.



We could use feature branches for each patch/patchset and do a merge in the main branch. That would be how upstream development works for patchsets, but some of our patches are rather long-lived for various reasons. Doing merges like this over time would lead to a very tangled history and it wouldn't really provide much benefit to anyway.



Another way would be to start with a base version, add our patches, and then merge in the upstream kernel changes on top of that. This is essentially the opposite of the scenario above. It still results in a rather tangled history over time, and it actually sends the wrong message: this git tree isn't an upstream. It's supposed to be a downstream representation of what we've added/changed. We don't want people developing new kernel drivers/features against the Fedora git tree. We want them to develop against upstream.



Another factor in figuring this all out was our build system. As I mentioned, koji isn't setup to take random exploded source git trees as build input. It expects things from particular places and in particular ways, which is perfectly fine. To make this git tree a "source" for koji, we'd have to use it as a snapshot tree and not list out the patches and such in the spec file. There are distros that do this, and maybe that's suitable for them but I find it valuable to look at the SRPMs from a distro and be able to easily see what's changed [sideways looking emoji].



So in the end, that means the git tree became an exact mirror of what happens to the kernel RPM in Fedora. Namely, it rebases on top of whatever upstream is doing. Now typically this is a terrible thing to do to downstream consumers. It means they need to force update their tracking branches and such. I'd be very hesitant about it if not for the fact that it's mostly just an ease of use thing. Another point is that it's analogous to how linux-next operates (albeit for different reasons). Overall, I think this is the best trade off between "git tree exists" and "maintainer remains sane". The follow section should help illustrate why as well.



How it works: all your rebase are belong to us

How it works is pretty simple. The tree tracks the main upstream git repo in the master branch, and the Fedora content for a release is contained on the "rawhide" branch. As of right now, only the rawhide kernels are tracked here, mostly due to time but also somewhat to see if this whole experiment is worthwhile. So when we go to update the Fedora kernel package in rawhide (which is typically a daily build), we do:



git remote update; git checkout master; git merge origin/master; git checkout rawhide; git rebase master;



That sequence pulls down the latest from Linus onto the master branch, then we rebase the rawhide branch on top of the changes we just pulled. We fix up any issues during the rebase, and then we're set.



At this point we could call it good and just push it to the remote repository, but that didn't really feel worthwhile. I wanted to gain something out of this as well, so I started thinking about how I could actually use the tree. I came up with a couple of things.



Since we're rebasing the patches in git, we don't need to do it separately in the Fedora package repo. There's no sense in doing work twice. After some initial renaming of some patches and such, I now use the git tree to generate the patches we add to the spec file by using git format-patch master.. and a script to copy them to the working dir on my machine. This means we always have a nice fresh copy of the patches for that specific upstream base. It does mean that each patch typically gets one line of change (the sha hash of the commit) everyday, but I don't think that's a big deal. This actually saves me time now and it helps keep our patches fairly "clean". They all apply with git-am and most of them have changelogs and such. An overall improvement from before.



I thought the tree itself could still be more useful to people as well. As the last commit of each update, I include the generated kernel configs that we build the kernel with. Our config setup in the package is somewhat confusing to people that don't work with it daily, and it's not obvious how all the fragments go together. This has them all in one place.



Then I realized that isn't particularly helpful without a way to track things (remember, we rebase every time). To fix that, I add an annotated git tag for every build we do in koji. The annotation points to the build task URL in koji itself. Someone that wanted to could browse to a particular tag and have a directly link to the Fedora package the source represents. You can also diff between tags, etc. I added the tags starting around the 3.17-rc4 builds.



(IMPORTANT: the link does not mean that package was built from this git tree. It simply points to the build that best represents it. Official builds are done from the Fedora package repo via SPEC file and tarballs/patches as mentioned before.)



With those things in place, we push out the results to the master and rawhide branches for the day, and call it good.



But what about when other Fedora maintainers add, remove, or modify a patch? Thankfully that happens less frequently than most people think. When it does, git rebase -i is my friend. It lets me rebase the patches while editing or amending, etc. If you haven't used it before I would highly recommend learning how it works.



Whither yon git tree?

OK, OK. You've probably had enough of my blabbering so you can find the git tree here:



https://git.kernel.org/cgit/linux/kernel/git/jwboyer/fedora.git



What's next?

To be honest, I'm not sure. The obvious thing would be to add the release branches (F20 and F21. F19 is dead to me.) so that will probably happen. Beyond that, I don't see much changing at this point. I'm very curious to see if people use this, and if they use it for something more than a web viewer of the tree. Time will tell!



Overall I'm glad I went through the exercise. It's been informative and beneficial, even if only in a small way.



TLDR:

What: exploded source git tree for Fedora kernel package (rawhide only for now)

How: rebases with every push; annotated git tags pointed to corresponding builds

Where: https://git.kernel.org/cgit/linux/kernel/git/jwboyer/fedora.git

Next: Up to you mostly!