gittup.org

A new linux distribution!

This is an entire(-ish) linux distribution in git. Everything is built with tup. That's why it's called gittup. You can find it on github here.

How it works

A linux distribution is just a collection of packages. Each package has a development history:

For example, the green tree might correspond to Linus' kernel tree. Your typical linux distribution will select particular versions of the packages and put them all together. As a user you end up with something like this:

With gittup.org, you get the whole development history (where applicable) as part of the standard distribution. As a user you get this:

All of these git projects become submodules of the gittup.git repository. The extra yellow/purple/orange nodes correspond to the gittup branch that I maintain at gittup.org. This mostly includes the changes to support tup, which are needed to make configuration changes and bisection across the entire distribution more practical. Generally what I do is:

Add Tupfiles in place of Makefiles

Add Kconfig support to configure the software (if not already present)

Remove .gitignore files (they are generated automatically by tup)

Sometime later if I want to update the software to a newer version, I'll pull from the upstream repository and merge it to the gittup branch:

Of course, since you have the git history of all the packages, you can do it too. If it all works, you can send me the patch to include support for the new version.

What's included

You get the git trees of:

alsa-lib from git://git.alsa-project.org/alsa-lib.git

alsa-utils from git://git.alsa-project.org/alsa-utils.git

binutils from git://github.com/jiez/binutils.git

busybox from git://busybox.net/busybox.git

linux from git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

uClibc from git://uclibc.org/uClibc.git

Additionally, you get releases of:

mplayer and supporting libraries (imported from svn)

openssh-5.2p1

openssl-0.9.8k

sshfs-fuse-2.2

gcc-core-4.4.1

ncurses-5.7

nethack-3.4.3

zlib-1.2.3

These are all imported into git from their respective tarballs. I couldn't find actual version control repositories for ncurses or nethack, which is why those have to be imported. I started to use gcc from their git repo, but the newest gcc in git is like 50% slower for some reason, so I ditched it along with all their other crazy non-C compilers and just imported gcc-core.

How to install

To bootstrap everything, you'll need some development tools. Make sure you have at least:

gcc

ar

ld

nm

objcopy

flex

bison

gawk

ncurses development libraries (for kconfig)

gperf (for kconfig)

mpfr development libraries (may be included in gittup in the future)

gpm development libraries (may be included in gittup in the future)

zlib development libraries (may be included in gittup in the future)

qemu (if you want to run it in an emulator)

apt-get install build-essential flex bison qemu gperf libncurses5-dev libncursesw5-dev gawk libmpfr-dev libgpm-dev zlib1g-dev yasm graphviz

Install tup

You need tup, so install that and put it in your path somewhere.

Install kconfig

You also need to build a patched kconfig, which supports the pushd/popd directives. This is so I don't have to modify the Kconfig files in the kernel or busybox or wherever.

~$ git clone git://gittup.org/kconfig.git ~$ cd kconfig kconfig$ tup init kconfig$ tup upd

And make sure you add tup-mconf and tup-conf to your path as well.

Download gittup.org

Now you need to download the entire distribution. This is about 600MB worth of stuff to download, so it'll be a while. Also you have to type "git" and "tup" a few million times, because everything involves either git, or tup, or is named after git and/or tup.

~$ git clone git://github.com/gittup/gittup.git ~$ cd gittup gittup$ git submodule init gittup$ git submodule update

Once you get to this point, you own gittup.org as much as I do. Well, I still own the domain name, cuz I ponied up like 10 bucks for that, but I mean you have all the source, all the history, even this crappy webpage. You can post it all online somewhere and start your own linux distribution.

Building gittup.org

Anyway, now you need to setup tup and compile everything. You need to initialize tup once. This sets the root of the build directory where tup will watch for files and track dependencies. It's kinda like git init, but for the build system instead of the version control system.

gittup$ tup init

Now you need to configure the distribution. This is done with Kconfig, which you run by using ./menuconfig or ./oldconfig. The first time you run menuconfig, please set the following configuration options:

ON : Linux Kernel -> General Setup -> Initial RAM filesystem and RAM disk (initramfs/initrd) support

: Linux Kernel -> General Setup -> Initial RAM filesystem and RAM disk (initramfs/initrd) support ON : Linux Kernel -> General Setup -> Configure standard kernel features

: Linux Kernel -> General Setup -> Configure standard kernel features OFF : Linux Kernel -> General Setup -> Configure standard kernel features -> Load all symbols for debugging/ksymoops

: Linux Kernel -> General Setup -> Configure standard kernel features -> Load all symbols for debugging/ksymoops ON : busybox -> Coreutils -> mknod

: busybox -> Coreutils -> mknod ON : busybox -> Init Utilities -> init

: busybox -> Init Utilities -> init ON : busybox -> Shells -> Choose your default shell -> ash

: busybox -> Shells -> Choose your default shell -> ash ON : gittup.org initial ramdisk -> Build initial ramdisk

gittup$ ./menuconfig

You can also just start with the example config and tweak it as you like:

gittup$ cp example-tup.config tup.config

Finally, you can build everything. You can use 'tup upd -j2' or something if you want to parallelize the compilation. Tup is parallel-safe, which is totally sweet.

gittup$ tup upd

Whenever you edit source files, Tupfiles, change the configuration, pull things from git, or whatever, just run tup upd to bring everything up-to-date.

If you want to avoid the time to scan the filesystem everytime you run tup upd, you can run tup's built-in file monitor (this uses the inotify interface). If you restart your computer, the file monitor needs to be restarted as well. You may need to up the /proc/sys/fs/inotify/max_user_watches variable before doing this. Sometimes it works for me with the default 8192, and sometimes I just bump it to 65536.

gittup$ tup monitor

Although the shell returns immediately after the monitor command, tup will be constructing the initial filesystem database in the background. It finishes when you see the line "Initialized in X seconds."

With the monitor running, you should be able to change any file in the system and have tup upd start recompiling within a few milliseconds. Of course, depending on the scope of the changes, the actual build time may be much longer.

Running in qemu

Use the qemu.sh script to run the kernel and initrd that you just built. There is no root filesystem image, since everything runs from the ramdisk.

gittup$ sh qemu.sh

When it boots, ncurses generates some weird messages about /bin/terminfo.src, but after a bit you should be able to press enter to get a shell. Then type nethack. First one to ascend in nethack using gittup.org wins an autographed CD with all my favorite MODs. I know you're thinking: "How can I pass up a CD of crappy music, autographed by some guy I don't know, who didn't even write the music?" I couldn't express it better myself. Now get to ascending!

Saving your config

You should save your tup.config file so you can easily rebuild the distribution you just built. Probably the best way to do that is to track it in git. If you want to you can track it in a separate branch and merge the gittup.git master branch into your local branch. Another option is to just check it into the master branch. It shouldn't ever conflict, since I won't push a file called "tup.config" into the git repository. However, if you use the master branch, you will get messages like "Your branch is ahead of origin/master by X commits", which may be annoying if you plan to do real development in gittup.org and send me patches.

gittup$ git add tup.config gittup$ git commit

Feel free to now play around with the config, but keep in mind that kallsyms and loadable kernel modules are not yet supported. Other options that I don't use may be broken as well, since my script to convert Makefiles to Tupfiles is not perfect.

Regular updates

Periodically you may want to update to the latest version. This involves getting the new changes onto your machine with git, and then building them with tup.

gittup$ git pull gittup$ git submodule init gittup$ git submodule update gittup$ ./oldconfig gittup$ tup upd

What gittup.org does that nobody else can

Make changes!

Harness the awesome power of tup to make changes to your system -- anywhere in the system -- and quickly see the results.

[marf@captainfalcon gittup]$ vi nethack/src/spell.c [marf@captainfalcon gittup]$ vi busybox/coreutils/ls.c [marf@captainfalcon gittup]$ time tup upd -j2 Executing Commands [ 0/9 ] busybox/coreutils/CC ls.c [ 1/9 ] nethack/CC src/spell.c [ 2/9 ] busybox/coreutils/LD built-in.o [ 3/9 ] busybox/LD busybox [ 4/9 ] nethack/LD nethack [ 5/9 ] initrd/bin/CP busybox [ 6/9 ] initrd/bin/CP nethack [ 7/9 ] initrd/MKINITRD [ 8/9 ] initrd/GZIP initrd [ 9/9 ] real 0m1.571s user 0m1.888s sys 0m0.269s

Yeah, I'm marf. Yeah, my computer is captainfalcon. Yeah, I just edited the spellcasting in nethack because I felt like it. Then I changed ls to print "Sup bro, way to list those files", because I like to be encouraged when I list things, and because I like it when people call me "bro"1. Then I re-compiled both and got a new initrd in like two seconds. Go ahead -- try to change the ls on your system to print out extra messages for no reason!2 You can't do it!! Unless of course you're running gittup.org. But if you're running gittup.org, why aren't you playing nethack or needlessly recompiling things just for fun? In fact, how are you reading this webpage?? It doesn't even come with a web browser.

1Not actually true.

2Naturally, this is a metaphor for doing something useful, and having a real build system to make it possible.

Note: If you harmlessly change a file in gcc, it still takes forever, since it recompiles everything, since I don't have tup short circuit builds, yet.

Note: You'll probably want a decent working knowledge of git to manage your changes. You can probably use format-patch to send me patches, and I think I can include them with 'git am' somehow. However, I'm no GitSu master, so I'll probably screw it up and have to re-do everything like always.

Bisection!

Let's say your compy86 is running fine, and then you update to get the latest, and something breaks. You can use git bisect across your *entire* linux distribution to find the commit that broke it.

Note: It may not work across config changes, since tup doesn't automatically run 'oldconfig' when necessary.

Note: I've never actually tried this, so we'll just assume it works.

What gittup.org doesn't do

In fairness, there are a few things gittup.org doesn't do. For example, you might use it to browse the web. Well, gittup.org can't do that. In fact, all you can really do with gittup.org is recompile things really fast and play nethack and watch movies with mplayer over sshfs.

Some things that actually need fixing:

Support arches other than x86 (I hardcoded some things... lazyness)

Kallsyms support in the kernel

Loadable kernel modules

Some things that will never be in gittup.org: