build2 | FAQ

Who is behind this?

Code Synthesis, Boris Kolpackov, and some friends. And maybe you?

Why are you doing this?

To make something people want? No, scratch that: we are making what we need.

We have a bunch of cross-platform C++ tools and libraries that are becoming an ever increasing pain to develop, test, distribute, and support. A mailing list thread of 20 messages where we painstakingly discovered that the user accidentally built our library with libstdc++ and his application with libc++ which all resulted in mysterious crashes (exception handling was not working properly) – this is a waste of life.

And while on a philosophical subject, note that we are not trying to create a wonderful unicorn that will magically solve all your building and packaging issues. Rather our aim is something like git – an engineering tool that allows serious developers working on complex projects to get stuff done without too much cursing. And if you think git is a wonderful unicorn, try to use submodules.

What does 2 in build2 stand for?

It stands for take two, after a previous, GNU make -based attempt called build .

Why did you call the build system driver b ?

We found typing something like make -j8 too tedious so we've decided to be bold and call it just b (the -j8 part is unnecessary since the hardware concurrency is detected automatically).

In retrospect, this was probably a bad idea since, somewhat surprisingly, it seems to upset some people. Yes, in the UNIX world, single letter commands are reserved for user aliases (though in my GNU/Linux /usr/bin/ I have w , X , and, of course, [ ). Note, however, that we often have to run b on machines (like build bots) where one is unlikely to bother with aliases. And it would be nice to have the same short name available everywhere.

In any case, it is probably too late to change now (maybe in version 2). Also, if you really must, you can always add an executable prefix or suffix when installing build2 .

What is wrong with <your favorite build system>?

In short, it doesn't provide a well-defined, consistent build interface across all the platforms/compilers out of the box (and no, having to install and, more importantly, maintain Python, Java, MSYS2, etc., doesn't count).

Also, as languages like Rust and Go clearly show, a build system alone is no longer sufficient: to handle the entire project development lifecycle we need a well-integrated build system and dependency manager.

In addition, the following areas are often poorly supported (or outright broken) in many existing (and new) build systems:

Development & Distribution Trying to support both is often tricky since they pull the design in different directions: for development we want power and flexibility while the end-user wants the simplest thing that does the job and gets out of the way (they definitely don't want to install anything extra). Black Boxes Many existing build systems are black boxes, they don't have a mental model of how the build is described and performed. The problem with this approach comes when you need to build something different: your only option is to start creating another black box. In this sense build2 is a lot like make (it has a notion of targets, prerequisites, rules, etc.) rather than, say, CMake (which has a notion of, uh, hm, magic incantations, maybe?). Reliable Builds We often find ourselves doing complete rebuilds for good measure because we don't trust our build system to track all the dependencies (compiler upgrades, option changes, etc). In build2 we do High Fidelity Builds. Cross Compilation This should be straightforward if you design for it from the ground up but many build systems just can't get it right. In build2 it is as easy to cross-compile as to compile natively. Source Code Generation Automatic header dependency extraction combined with auto-generated source code can get tricky and many build systems have given up on handling this properly and provide make-shift solutions such as pre-build steps. In build2 this is part of the build model. Project Importing/Composition In build2 we have an explicit model for importing one project into another (e.g., a project imports a prerequisite library, including installed) as well as for amalgamation (you can drop a prerequisite library into your project and the import machinery will automatically pick it up). Extra Operations These days build systems rarely handle just building. There is also configuration (often separate/add-on but integrated in build2 ), testing, installing/uninstalling, and preparation of distributions. Toolchain, not just a Tool In build2 we have a tightly-integrated dependency manager that informs the design of the build system (and vice versa; though the build system can be used by itself, without any packaging). Sane Syntax While at it, why not have a nice, clean syntax (no scripting language extensions, etc)?

But what if you are only interested in one platform, stay away from code generators, and like opaque boxes? It's true, if you develop, say, a Windows-only application in Visual Studio you will most likely find whatever comes with your IDE a lot more convenient. However, you may still find build2 useful if you have dependencies on third-party libraries: you can use the toolchain to build them for your exact configuration(s) and keep up with their updates.

How is this better than CMake (or Meson)?

While most of the general issues described in the previous question apply to CMake and more specific ones are listed below, here is a TL;DR of practical advantages that all this translates to:

Uniform: works the same everywhere, no project generation step.

Sane syntax with wildcard patterns support.

Support for C++ Modules.

Support for Cross-compilation.

Precise change detection: skips recompilation of ignorable changes (comments, whitespaces, etc).

Well-integrated toolchain (dependency management, etc), not just a build system.

Dependency-free: all you need is a C++ compiler.

To expand on that, the core problem both build2 and CMake try to address is this: there are all these different platform/compiler-specific build systems out there and it is painful to support them individually. The CMake approach is this: let's come up with a single project description that we can translate to all those underlying build systems. In contrast, the build2 approach is this: let's come up with a single, uniform build system that we can use on all the platforms and with all the compilers (and if we do a good job, then hopefully IDEs will start using it as well).

The nice thing about the CMake approach is that you can reuse all the existing build systems for doing actual building. Plus people can continue using, for example, Visual Studio since they get its project files (but they have to regenerate them and reload if they add/remove any files).

The bad thing about the CMake approach is that it is a race to the bottom, to the lowest common denominator. If you want to support multiple platforms, you can only use build system features that are available on all of them. This gets especially hairy when it comes to supporting cross-compilation, source-code generation (and especially header generation; see this thread for details), as well as providing additional operations like install , uninstall , dist , etc.

Another general problem with CMake is the lack of full control, all the way down to running compilation commands. A good example of where it hurts is support for the Clang Compilation Database which is used by static analysis tools, editors for auto-completion, etc. CMake simply cannot generate it for Visual Studio builds because it doesn't know the actual compilation command lines that Visual Studio will use. In a sense, this build system is often one black box ( CMake ) stacked on another (Visual Studio/MSBuild).

Let's now see what the build2 approach gives us:

Now we have a uniform build system interface that works across all the major platforms (Linux, Mac OS, Windows, FreeBSD, etc.) and compilers (GCC, Clang, MSVC, Intel, etc).

We can handle source code generation properly, again, for all the platforms/compilers.

We can do cross-compilation, properly. In build2 we just don't do anything that is not "cross-compile"-clean, end-to-end.

we just don't do anything that is not "cross-compile"-clean, end-to-end. build2 provides a set of operations that you would expect from a modern build system, again, uniformly and across all the platforms/compilers: configure , update / clean , test , install / uninstall , and dist (prepare distribution archives).

provides a set of operations that you would expect from a modern build system, again, uniformly and across all the platforms/compilers: , / , , / , and (prepare distribution archives). And as a bonus you get well-integrated dependency management which again works the same way everywhere.

Why does Ninja start compiling faster than build2 ?

TL;DR: While there may be an initial moment of "silence" when building with build2 , useful work is being done and the overall build might even be slightly (because of file caching) or even significantly (because of more precise change detection) faster. If you find the lack of any output unnerving, use the -p|--progress option. If your build machine is nearby you may also find the sound of fans spinning up comforting.

Ninja and build2 use different C/C++ compilation models when it comes to handling header dependencies (sets of headers each translation unit depends on that are used to decide what needs to be rebuilt).

Ninja extracts header dependencies as part of the compilation step itself. This allows it to issue the first compilation command pretty much immediately but requires all the headers to already exist and be up-to-date. Which means no auto-generated headers, at least not as part of the overall build (maybe as a pre-build step).

In contrast, build2 extracts header dependencies before starting the compilation. While it means there is a delay before you see the first compilation command, this approach has a number of advantages: It supports auto-generated source code as part of the build itself which can become essential for complex code generators like ORMs that may need to extract their own dependencies. This setup also opens the door for supporting C++ modules which in many ways are like auto-generated headers: the build system has to update all the binary module interfaces before it can start compiling a translation unit that imports them.

And this is not all: there are even more advantages but to fully appreciate them we need to understand how build2 sets this compilation pipeline up. At the compiler level, header dependency extraction is essentially a preprocessor run: it has to process #include directives recursively (and keeping track of which ones are #ifdef 'ed out, etc.) and produce the list of headers that have been included. So instead of extracting only the header list, build2 instructs the compiler to also save the (partially) preprocessed output. Later, if the translation unit needs to be rebuilt, the compiler is given this (partially) preprocessed file rather than the original.

So build2 does do some extra work compared to Ninja but the results of this work are not wasted. While one would still expect Ninja to be faster overall (one compiler invocation rather than two, no extra filesystem access to save the preprocessed file, etc.), the reality is surprising: in certain cases the build2 approach results in a slightly faster overall build. The reason is probably the time-localized and cached access to all the headers; with build2 all this happens at the beginning of the build while with Ninja the access is spread over the entire build. See Separate Preprocess and Compile Performance for details.

Going back to other advantages, now that we have a (partially) preprocessed translation unit we can do a few more interesting things: we can perform more precise change detection by ignoring changes that do not affect the resulting token stream (or its debug information) such as changes to comments, macros that are not used, etc. In build2 this precise change detection is combined with the module dependency (import) extraction.

Finally, the (partially) preprocessed unit is a perfect candidate for shipping to a remote machine for distributed compilation. Since all the #include directives have been expanded we don't need to worry about the remote machine having exactly the same set of headers in exactly the same locations.

Note also that build2 itself is multi-threaded: it not only runs compilation jobs in parallel but also parallelizes internal tasks, such as parsing of the dependency output, calculation of the translation unit checksums, etc.

Do you support cross-compilation?

Yes, cross-compilation was supported from day one with native compilation being just a special case (build is the same as host). Fun fact: build2 could cross-compile to Windows before it could build natively.

What is wrong with using system package managers?

In short, there is nothing wrong with them, they work well (unless you are on Windows), and we use them ourselves every day. And build2 (both the build system and package manager) work well with system-installed packages.

To elaborate, system package managers (such as rpm , dpkg , FreeBSD pkg , etc.) serve a different purpose compared to bpkg and bdep : they are optimized for the end-user who wants to easily install and use a program, not the developer who wants flexibility and power in juggling their library/tool dependencies. Yes, most system package managers support installing development files for a library but you can normally only have one version at a time. While during development we often want multiple versions (so we can make sure our code works with all of them) that are built in multiple configurations (different compilers, 32/64-bit, debug/release, and don't forget cross-compilation). And this is what build2 is optimized for.

Still, it often makes sense to use a system-installed library or tool instead of building it from source. Take libsqlite3 as an example: it is stable, available and works the same way pretty much everywhere – it probably doesn't make much sense to build it from source for every configuration.

The build2 build system will happily use a system-installed library; there is even pkg-config integration to extract additional information such as the location of headers, additional libraries to link, etc. Also, the package manager has a notion of system-installed packages (that sys: prefix) which allows you to instruct bpkg and bdep to assume the package is available from the system rather than building it from source.

What makes you think you will succeed when so many have failed?

Good question. For over 10 years now we've been developing and using in production our own, GNU make -based build system. On several occasions we've also contributed new features to GNU make (second expansion in pattern rules, shortest stem matching, etc).

Despite being based on the build tool not designed for today's complexity, build is surprisingly powerful. For example, build can generate (based on templates) other build systems, such as autotools and VC++ projects/solutions. We call this meta-build and it's the reason why most people who use our projects have never heard of build .

However, the original make design and GNU make codebase do show their age. The biggest problem is the lack of out of the box Windows support. Or, more precisely, the lack of a POSIX shell on Windows. Also, the interaction of complex features like parallel builds, auto-generated source code, and automatic header dependency extraction can lead to order-dependent behavior. For example, a C++ preprocessor is sometimes executed on a header file as it is being generated. We euphemistically call them build aberrations and they will not be easy to fix in make . But, despite all of this, many design decisions that you will find in build2 have been conceived and battle-tested first in make and build .

When it comes to the package manager, our view is pretty simple: given a sane build system with a consistent interface across platforms, writing a package manager is actually not that hard. The core functionality of bpkg was designed and implemented in a month. However, try to support different/multiple/insane build systems and things get hairy fast.

Code Synthesis also has a solid track record of making complex C++ tools work across multiple platforms/compilers, including cross-compilers (see ODB for a good example). So, to summarize, we have experience developing build tools/systems and using them to deliver real-world, cross-platform C++ products. But then, again, none of this is a guarantee. We will just have to wait and see.

Why did you write it in C++ instead of <your favorite language>?

Let's see: we need a cross-platform (must work out of the box on Windows) and reasonably efficient (nobody likes to wait for their up-to-date checks) language that most C++ developers already know (so that they can extend build2 if they need to). Any language comes to mind that fits the bill? Plus, nothing gives as good a reality check as trying to build your own build system and package your own package manager.

There is also this aspect: developing a build system and/or package manager is a multi-year, full-time project. If you are writing it in, say, Python, then this means you are a full-time Python developer trying to design and implement a major piece of C++ infrastructure for a language you don't actually use yourself for any serious development.

Why aren't you hosting on GitHub?

Good news: most of the build2 project repositories are now mirrored on github.com/build2 and you are welcome to submit issues, pull requests, etc., via this channel. There is also the #build2 channel on cpplang.slack.com (get an invite) where bug reports are welcome as well.

We still host the master repositories ourselves because we want complete control of our infrastructure. Also, many things on GitHub are optimized for the convenience of new/inexperienced git users and not for the efficient handling of large volumes of information by the experienced ones. Case in point: issue tracker; I am much more efficient using my mail client ( mutt ) and text editor ( emacs ) than a web browser form.

Also, if this setup works for the Linux kernel, surely it should work for us; see the "Why kernel development still uses email" LWN article for details.

How do I submit bugs/patches?

You can either email them to the users@build2.org mailing list or create an issue/pull request on github.com/build2 . There is also the #build2 channel on cpplang.slack.com (get an invite) where these are welcome as well.

While you can just fire and forget (we do realize that by submitting a bug report or a patch you are doing us a favor, so we are not picky), we would appreciate it if you took a look at the posting guidelines. Note that to post to the mailing list you don't need to subscribe.

What are your plans for cppget.org?

We hope it will become The C++ Package Repository, the place where people publish theirs and look for other's C++ libraries and tools. If/when this happens, it will probably be a good idea to turn the control over to someone impartial, like Standard C++ Foundation (to this effect, we are running build2.org and cppget.org on completely separate infrastructures from the start).

Do you plan to support binary packages?

Not at this early stage but maybe in the future.

The problem with supporting binary packages is you have to make sure the binary package built on someone else's machine matches your environment precisely (if you think this is a Windows-only problem, check the recent GCC dual ABI mess). If we start cutting corners here we will end up in tears (and long mailing list threads). One interesting approach would be to dump the entire build system state into, say, an SQLite database and then ship that with the binary package. On the other end we can then load this database and verify that all the prerequisites (like system headers, compiler versions, etc.) match.

Also, there are some ideas floating around about making bpkg automatically generate binary packages in rpm and deb formats. After all, the bpkg manifest already includes all/most of the information needed.

Can I run my own private/public repositories?

Yes, the entire toolchain, including the server-side web interface, is open source, licensed under the MIT License.

How are you going to make money from this?

We are not sure. Maybe we won't. Honestly, if this succeeds, and we won't have to manually test/distribute our tools and libraries on the myriad of platform/compiler/IDE combinations, then it would have already been worth it.