build2 | 0.12.0 Release Notes

These notes provide a more detailed discussion of major new features, including the motivation for implementing them and their usage examples. For the complete list of changes, refer to the Release Announcement or the NEWS files in the individual packages. See also the discussion of these release notes on r/cpp/ and r/programming/ .

The main objective of this release was support for build system modules and it took a while to tie up all the loose ends in this area. As a result, this release has accumulated quite a large number of other major new features across all the tools in the toolchain (build system, package and project dependency managers, etc). While the below somewhat long notes discuss most of them in detail, here is a one line TL;DR and a link for each feature:

A note on backwards compatibility: in this release the build system saw a major reorganization (discussed below). As a result, this release cannot be upgraded to from 0.11.0 and has to be installed from scratch.

In addition to the toolchain changes, the CI service now offers the following new build configurations:

linux_debian_10-gcc_9.2 linux_debian_10-clang_9.0 linux_debian_10-clang_9.0_libc++ macos_10.15-clang_11.0 macos_10.15-gcc_9.2_homebrew freebsd_12-clang_8.0 windows_10-msvc_16.3 windows_10-clang_8.0_msvc_16.3 windows_10-clang_9.0_msvc_16.3 windows_10-clang_9.0_msvc_16.3_lld

The total now stands at 40 build configurations and covers a wide range of versions for all the major compilers (GCC, Clang, and MSVC) on all the major platforms (Linux, Mac OS, Windows, and FreeBSD).

And in other news, we now have the Contributing section on our Community page as well as the Project Roadmap.

1 Toolchain

1.1 Local toolchain installation

By default the install scripts download source archives necessary to bootstrap the toolchain ( build2-toolchain plus, in case of Windows, base utilities) and build the final installation from packages obtained from the package repository ( cppget.org ). This allows upgrading to a newer version of build2 directly from packages and without having to go through the bootstrap process.

Now all the install scripts and batch files support the --local option to skip building the final version from packages and instead build it from the build2-toolchain archive directly. This reduces the build time by about half and also allows performing an offline installation by pre-downloading the necessary source archives as described in How to install offline? Note that in case of a local installation, an upgrade to a newer version involves performing the same from-scratch installation.

1.2 Default options files (aka tool config files)

All the tools in the build2 toolchain now support configuration via pre-defined files. However, instead of having a separate config file format, the build2 toolchain uses default options files which contain the same options as what can be specified on the command line.

The default options files are searched for in the .build2/ subdirectories of each outer directory beginning from the start directory until reaching either the home directory or the filesystem root and then in the .build2/ subdirectory of the home directory and finally in the system directory (for example, /etc/build2/ ) if configured.

Where exactly the start directory is as well as which files are searched for is defined for each tool. And, for some tools, like the project and package dependency managers – for each command (and, sometimes, even for each command's mode).

For example, most bdep commands start searching for default options files in the project directory. This allows us to customize the behavior of the bdep 's commands on the per-project basis and even commit them to the project's repository so that they can be shared between everyone working on the project. For example, to automatically use the .hpp / .cpp naming scheme for our source code we could add the .build2/bdep-new.options file with the following contents to our project's repository:

--lang c++,extension=?pp

See the b(1) (the DEFAULT OPTIONS FILES section) as well as the bdep-default-options-files(1) and bpkg-default-options-files(1) man pages for details.

2 Build System

2.1 Dynamically-buildable/loadable build system modules

A major chunk of build system work went into support for build system modules that are developed independently of the build system core. Such modules can be used to extend the build system to support new languages, source code generators, new operations, functions, target types, etc. In fact, build2 was designed from the ground up to allow external modules to customize pretty much every aspect of the build system functionality. For example, the configuration support is provided by the config module that is bundled with the build system core but if you wanted to, you could write your own and use that instead.

In a nutshell, a build system module is a separately-packaged shared library project with the libbuild2-<modname> name. When build2 sees using <modname> in a buildfile , it will build and load this library.

In this release we have the ability to dynamically update (if necessary) and load such modules. Plus, the CI service fully supports testing build system modules so that you can make sure modules that you develop work on all the major platforms and compilers. In the next release we plan to plumb this support through the package manager so that the whole process of downloading, building, and loading a build system module (as well as any tools, such as source code generators, that they may depend on) is completely seamless.

If you would like to start playing with build system modules, there is libbuild2-hello to get you started. Plus some folks have already begun experimenting with real modules: see the "Yacc module?" mailing list thread (continues here). Finally, see the Build system reorganization mailing list post for technical details on modules support.

2.2 Build system as a library and build contexts

Two other interesting features came from the work on build system modules: build system as a library and the notion of a build context.

Before this release the build system was all in the b executable. Now it is split into libbuild2 (build system core), a bunch of build system modules (for example, libbuild2-c , libbuild2-cxx , etc; called bundled modules) and the b executable (build system driver). With this arrangement, someone else (for example, an IDE author) can write their own build system driver with their own set of modules, build state loading logic, and so on. In particular, this could be a great way to experiment with the build-by-convention ideas, alternative build description formats, etc. See the Build system reorganization mailing list post for details.

And in order to support automatic building of build system modules we needed the ability to preempt the current build in order to build the module. To support this we have introduced the notion of a build context which contains all the non-const global state of a build system. As a result we can now have multiple independent builds at the same time.

2.3 Pattern matching ( switch )

If you are not familiar with pattern matching from other languages, an example is probably the best way to illustrate the idea:

switch $cxx.target.class, $cxx.target.system { case 'windows', 'mingw32' cxx.libs += -lrpcrt4 case 'windows' cxx.libs += rpcrt4.lib case 'macos' cxx.libs += -framework CoreFoundation }

One interesting feature of build2 's switch is the ability to specify the match function so that we can use other kinds of patterns in addition to straight comparison (regex, shell wildcard patterns, case-insensitive comparison, etc). For example:

switch $cxx.target.cpu: regex.match { case 'i[3-6]86' ... case 'x86_64' ... }

See Pattern Matching ( switch ) for details.

2.4 Clang targeting MSVC and MSVC installation discovery

This release adds first-class support for Clang targeting the MSVC runtime, including using LLD. In particular, this means the build2 toolchain itself can now be built with Clang on Windows (see build2-install-clang.bat ) and several Clang MSVC build configurations are now part of our CI service.

We were pleasantly surprised by how stable Clang's support for targeting MSVC is. 8.0.1 that ships with Visual Studio has been rock solid and we ran into only one mis-compile in 9.0.0 which has already been fixed for 10 and 9.0.1. What's more, LLVM binutils (LLD, LLVM-lib, etc) work well as replacements for Microsoft's versions and we even get thin archive support with LLVM-lib. And now that the MSVC runtime itself is open source, this opens up a real possibility for cross-compiling to Windows.

Note also that build2 uses the "straight" Clang drivers and not the clang-cl wrapper (but limited support for clang-cl is also provided). See Clang Compiler Toolchain for details.

As part of this work we have also implemented automatic installation discovery for MSVC 15 (2017) and later, similar to how Clang does it. In particular, this allows building outside the Visual Studio development command prompts. See MSVC Compiler Toolchain for details.

2.5 Configuration exporting and importing

The new config.config.save variable can be used to specify the alternative file to write the configuration to as part of the configure meta-operation. For example:

$ b configure: proj/ config.config.save=my-config.build

The new config.config.load variable specifies additional configuration files to be loaded after the project's default config.build , if any. For example:

$ b create: cfg/,cc config.config.load=my-config.build

Both config.config.load and config.config.save recognize the special - file name as an instruction to read/write from/to stdin/stdout, respectively. For example:

$ b configure: src-prj/ config.config.save=- | \ b configure: dst-prj/ config.config.load=-

The config.config.load also recognizes the ~host special configuration name. This is the default host configuration that corresponds to how the build system itself was built. For example:

$ b create: tools/,cc config.config.load=~host

2.6 Compiler mode options as part of config.{c,cxx}

In this release we have extended the config.c and config.cxx variables' semantics to allow specifying compile mode options. Such options are not overridden by buildfiles and are passed last (after cc.coptions and {c,cxx}.coptions ) in the resulting command lines. Note that they are also cross-hinted between config.c and config.cxx . For example:

$ b config.cxx="g++-9 -m32" # implies config.c="gcc-9 -m32"

But:

$ b config.cxx="clang++ -stdlib=libc++" config.c=clang

3 Project Dependency Manager

3.1 Creating new projects with existing files

The bdep-new command now supports creating new projects with existing files. It also recognizes and handles the following existing filesystem entries in the project root:

.git -- assume VCS is already initialized and is Git LICENSE -- try to guess the manifest license from contents README.md -- try to extract the manifest summary line from contents

Overall, the idea is to streamline the workflow where we create a project on one of the hosting services (GitHub, GitLab, etc) and then initialize it with bdep-new .

Also, to this effect, specifying the project name is now optional and if omitted, the current working directory is assumed to be the project name.

Here is an example of this streamlined workflow:

$ # create project with LICENSE and README.md on a Git hosting service $ git clone .../libhello.git $ cd libhello $ bdep new -t lib

You can also see it in action in the CppCon 2019 Why is C++ so Slow? lightning talk.

4 Package Dependency Manager

4.1 Specifying full version constraint on the command line

The pkg-build command now allows specifying the full package version constraint on the command line. For example:

$ bpkg build "bar < 2.0.0"