Redo from scratch [Jan. 12th, 2011|08:28 pm] pozorvlak

I think that djb redo will turn out to be the Git of build systems.



What do I mean by that? Well, think about version control systems before the cryptographic-DAG systems, and especially Git, came along. Everyone (everyone you'd want to work with, at least) agreed that a version control system was an essential part of a self-respecting coder's toolchain, but all the systems available sucked. Managing your VC repository required effort and care, and a lot of coders (myself, I'm ashamed to say, included) just learned the basics because nontrivial version-control tasks just got so complex, so fast, it was usually quicker to go outside the system or even redo your work. Even thinking about what was going on was hard. Large teams sometimes had someone whose entire job was to nursemaid the version control system. Clearly version control was a Serious, Complex Problem. Our best hope was a bunch of guys with serious beards who were developing an impenetrable mathematical formalism for attacking the problem; unfortunately, their code was far too slow, and their formalism at best delayed the onset of head-breaking complexity.



Then Git came along and blew away the old order. Once you've grokked the Git worldview, previously intractable problems dissolve before your eyes. It turns out that version control can actually be simple. There was a bit of kicking and screaming, but the open-source community is well on the way to adopting Git as a de facto standard for version control - removing, as a side-effect, one annoying hurdle to making your first contribution to any given open-source project. There are still holdouts, who use systems which do some things differently to the way Git does them; this is because they haven't yet realised that Git's way is right. Git is, quite simply, the Right Thing.



Now think about build systems. Everyone (who you'd want to work with...) agrees that a build system is an essential part of your toolchain, but all the available systems suck. Make, in particular, has three main problems: It's Yet Another Goddamn Syntax you have to learn, with its own stupid quoting and whitespace rules. Recursive make sucks. Nonrecursive make sucks. There are numerous make-replacements and make-addons and Makefile generators and they all add complexity, often for little obvious benefit: many of them just fix things that weren't broken in the first place.



I've spent a few hours over the last couple of days reading about and playing with redo , an implementation by Avery Pennarun of a design by the famously uncompromising genius Daniel J. Bernstein. And I've been feeling the same excitement as I felt when I started to understand Git. Redo has an incredibly simple design, but that design makes previously complex problems dissolve. I was initially sad to see that it didn't have fabricate.py's cross-language automatic dependency detection, but then I thought for a few minutes and realised that it would be trivial to add that to your build. If that's what you wanted, of course - core redo (unlike core make) is very, very small, but the community is planning to provide a library of commonly-desired features which you can turn on if (and only if) you want.



So I seriously hope that redo will do what Git did and take over the world, to the benefit of all. But it faces some obstacles.



Firstly, it's closely tied to Unix, both in its implementation and in its use of shell scripts as the default build-script language. From the documentation, it's not even clear if it's been tested on Windows. More recent versions of redo allow you to write do-files in any language that supports the #! convention, so writing cross-platform build scripts won't be that hard, but it will cause additional friction: shell, IMHO, is mostly a good language for this kind of thing. Pennarun talks about linking a POSIX shell directly into redo to provide Windows support, but that kinda misses the point - any nontrivial shell program typically relies on a host of standard Unix utility programs.



Git also suffered from poor Windows support early on, but in Git's case the Unix-centricity was less fundamental.



Secondly, there's the migration problem. It was possible to write importers and exporters allowing conversion between different VCSes, so established projects could move over to Git without losing history. Is it even possible to do that with a build system of any complexity? Redo has been carefully written so it can coexist with make, allowing piecemeal conversion, but that conversion will always, AFAICT, be a manual process. The mailing list is full of success stories about astonishing code savings made by moving to redo, but, again, someone has to understand the existing build system and write the new one. Hopefully the redo community will build up some expertise at handling migrations as time goes on - the redo project is only nine weeks old!



Thirdly, there's the fact that everyone already knows make. One Reddit user asked "What features make up for this not being make(1)?" The clear implication being that make is the standard, everyone already knows it, and redo has to be pretty exceptional for make not to be a better choice.



But does everyone really know make, or do they just kinda know the basics, like everyone used to "know" Subversion? Let me tell you a story. About six years ago, when I was working for a Dilbert-esque systems integrator, I was sent on a training course to learn about the new (and, as it turned out, spectacularly bad) version control system that my team was being ordered to adopt. Everyone in the room was the build manager of their respective project. The instructor asked us "OK, who's used make?" We all put our hands up. "Now, who's written a Makefile from scratch?" Only my hand remained up.



I confidently predict that if the portability and migration problems can be solved, then the community will fall on redo like a pack of starving wolverines.



[I gave a short talk about redo at the January 2011 Glasgow.pm technical meeting. My notes are available here.]