Goa - streamlining the development of Genode applications

The development of applications for Genode used to require a lot of learning about Genode's way of organizing source code, our custom build system, and the use of run scripts. With Goa, I introduce a new tool that aims at largely removing these burdens from application developers.

In contrast to the tools that come with Genode, which were designed for developing complete systems, Goa is focused on the development of individual applications. In a nutshell, it streamlines the following tasks:

The porting of 3rd-party software to Genode, which typically involves Downloading 3rd-party source code via Git, Subversion, or in the form of archives,

Applying patches to the downloaded source code, and

Keeping track of changes locally made to the downloaded source code. Building software using standard build tools like CMake, alleviating the need to deal with Genode's custom build system. Goa takes care of automatically installing of the required Genode APIs and supplying the right parameters to the build system so that Genode executables are produced. Rapidly testing the software directly on the developer's Linux host system. Goa automatically downloads Genode components needed for the test scenario. Since Genode executables are binary-compatible between Linux and microkernels, the same software as just tested on Linux can be deployed on top of the other kernels. Goa takes care of exporting the software in the format expected by Genode's package management. Publishing (archiving and cryptographically signing) the software so that it becomes available to other Genode users, in particular users of Sculpt OS.

A first example, using a plain old Makefile

Let's say, we want to build a hello-world application that uses the raw Genode API with no libc whatsoever.

First, create a project directory, let's call it "hello":

mkdir hello cd hello

By convention, the project name corresponds to the name of the directory. Source codes are stored in a src/ sub directory. Let's create a file at src/hello.cc with the following content:

#include <base/log.h> #include <base/component.h> void Component::construct(Genode::Env &) { Genode::log("Hello"); }

Besides the hello.cc file, let's create a Makefile at src/Makefile with the following content:

hello: hello.cc

Now, let's give goa a first try:

hello$ goa build

This command prompts Goa to create a new directory called var/ within the project directory. The var/ directory is the designated place for generated files such as the build directory. However, upon the attempt to compile the program, disaster strikes:

hello.cc:1:10: fatal error: base/log.h: No such file or directory #include <base/log.h> ^~~~~~~~~~~~ compilation terminated. make: *** [hello] Error 1 [hello:make] <builtin>: recipe for target 'hello' failed Error: build via make failed

Our program tries to include a header file that is nowhere to be found. To resolve this problem, we can tell Goa that our project needs to use the Genode base API, by placing a file named used_apis with the following content into the project directory.

nfeske/api/base

This line tells Goa that our project depends on Genode's base API that features the base/log.h and base/component.h headers. When issuing the command goa build again, we see the following message:

download nfeske/api/base/2020-02-27.tar.xz download nfeske/api/base/2020-02-27.tar.xz.sig

Goa automatically downloaded the base API for us and installed it into a fresh depot at var/depot/nfeske/api/base/2020-02-27. But not only that, it also re-attempted the build of the program. If we take a look at var/build/x86_64/, we see the hello executable. If the output was too unspectacular for your taste, you may append the –verbose argument to the goa build command to see more details about the steps taken.

To run the program, we need to tell Goa, which part of the build artifacts are relevant to us. In our case, it's the hello executable binary. We can declare this information in a file called artifacts with the following content in the project directory.

hello

If we issue the goa build command again, we can see that this file appears at var/bin/x86_64/hello. The content of the bin directory is meant for the integration into a Genode scenario.

Speaking of a Genode scenario, to run the program within a Genode system, we have to define the "contract" between the program and the surrounding system. This contract has the form of a runtime package. Let's create one with the name "hello":

hello$ mkdir -p pkg/hello

A runtime package needs at least two files, a README file and a runtime file. The README file should give brief information about the purpose of the Genode subsystem for human readers. The runtime file contains the contractual information. Create a file pkg/hello/runtime with the following content:

<runtime ram="1M" caps="100" binary="hello"> <config/> <content> <rom label="hello"/> </content> </runtime>

Here we declare the binary we want to start, how much RAM and capabilities the subsystem expects, configuration information passed to the subsystem, and the content of the package. In this case, we only have a single ROM module called "hello", which is our binary.

For running the scenario, we can use the goa run command:

hello$ goa run download nfeske/bin/x86_64/base-linux/2020-02-27.tar.xz download nfeske/bin/x86_64/base-linux/2020-02-27.tar.xz.sig download nfeske/bin/x86_64/init/2020-02-27.tar.xz download nfeske/bin/x86_64/init/2020-02-27.tar.xz.sig download nfeske/src/base-linux/2020-02-27.tar.xz download nfeske/src/base-linux/2020-02-27.tar.xz.sig download nfeske/src/init/2020-02-27.tar.xz download nfeske/src/init/2020-02-27.tar.xz.sig download nfeske/api/os/2020-02-27.tar.xz download nfeske/api/os/2020-02-27.tar.xz.sig download nfeske/api/report_session/2019-02-25.tar.xz download nfeske/api/report_session/2019-02-25.tar.xz.sig download nfeske/api/timer_session/2019-11-25.tar.xz download nfeske/api/timer_session/2019-11-25.tar.xz.sig Genode 20.02-1-gac1b2ec24e 17592186044415 MiB RAM and 8997 caps assigned to init [init -> hello] Hello

We can see that Goa automatically installed the dependencies needed to execute the runtime package, integrates a Genode scenario, and executes it directly on Linux. If you switch to another terminal, you can see the Genode processes:

$ ps a | grep Genode 8646 pts/3 Sl+ 0:00 [Genode] init 8649 pts/3 Sl+ 0:00 [Genode] init -> hello 8650 pts/3 Sl+ 0:02 [Genode] init -> timer

You can cancel the execution of the Genode scenario via Control-C.

A second example, using CMake

As another step, let us create a new project that executes the 2nd step of the excellent CMake tutorial. Let's call the project "cmake_step2". Instead of copying the code into the cmake_step2/src/ directory, let us better tell Goa to download the code from the original tutorial. This can be done by creating an import file in the project directory. Create the file cmake_step2/import with the following content:

LICENSE := BSD VERSION := master DOWNLOADS := cmake_step2.svn URL(cmake_step2) := https://github.com/Kitware/CMake/trunk/Help/guide/tutorial/Step2 REV(cmake_step2) := HEAD DIR(cmake_step2) := src

This import file uses Github's SVN bridge to download only the specified sub directory of the CMake project. Let's give it a try:

cmake_step2$ goa import import download https://github.com/Kitware/CMake/trunk/Help/guide/tutorial/Step2 import generate import.hash

After the command finished, we find the source code sitting nicely in a new src/ directory. Let's try to build it:

cmake_step2$ goa build [cmake_step2:cmake] -- The C compiler identification is GNU 8.3.0 [cmake_step2:cmake] -- The CXX compiler identification is GNU 8.3.0 [cmake_step2:cmake] -- Check for working C compiler: /usr/local/genode/tool/19.05/bin/genode-x86-gcc [cmake_step2:cmake] -- Check for working C compiler: /usr/local/genode/tool/19.05/bin/genode-x86-gcc -- works [cmake_step2:cmake] -- Detecting C compiler ABI info [cmake_step2:cmake] -- Detecting C compiler ABI info - done [cmake_step2:cmake] -- Detecting C compile features [cmake_step2:cmake] -- Detecting C compile features - done [cmake_step2:cmake] -- Check for working CXX compiler: /usr/local/genode/tool/19.05/bin/genode-x86-g++ [cmake_step2:cmake] -- Check for working CXX compiler: /usr/local/genode/tool/19.05/bin/genode-x86-g++ -- works [cmake_step2:cmake] -- Detecting CXX compiler ABI info [cmake_step2:cmake] -- Detecting CXX compiler ABI info - done [cmake_step2:cmake] -- Detecting CXX compile features [cmake_step2:cmake] -- Detecting CXX compile features - done [cmake_step2:cmake] -- Configuring done [cmake_step2:cmake] -- Generating done [cmake_step2:cmake] -- Build files have been written to: .../cmake_step2/var/build/x86_64 [cmake_step2:cmake] Scanning dependencies of target Tutorial [cmake_step2:cmake] [ 50%] Building CXX object CMakeFiles/Tutorial.dir/tutorial.cxx.obj .../cmake_step2/src/tutorial.cxx:2:10: fatal error: cmath: No such file or directory #include <cmath> ^~~~~~~ compilation terminated.

Apparently, the example requires the standard C++ library. We can supply this API to the project by creating a used_apis file with the following content:

nfeske/api/posix nfeske/api/libc nfeske/api/stdcxx nfeske/api/base

The posix API is needed because - unlike a raw Genode component - the program starts at a main function. The libc is needed as a dependency of the Standard C++ library.

When issuing the goa build command again, we see that Goa downloads the required APIs and successfully builds the example program:

cmake_step2$ goa build download nfeske/api/base/2020-02-27.tar.xz download nfeske/api/base/2020-02-27.tar.xz.sig download nfeske/api/libc/2020-02-19.tar.xz download nfeske/api/libc/2020-02-19.tar.xz.sig download nfeske/api/posix/2019-02-25.tar.xz download nfeske/api/posix/2019-02-25.tar.xz.sig download nfeske/api/stdcxx/2019-12-18.tar.xz download nfeske/api/stdcxx/2019-12-18.tar.xz.sig [cmake_step2:cmake] -- Configuring done [cmake_step2:cmake] -- Generating done [cmake_step2:cmake] -- Build files have been written to: .../cmake_step2/var/build/x86_64 [cmake_step2:cmake] [ 50%] Building CXX object CMakeFiles/Tutorial.dir/tutorial.cxx.obj [cmake_step2:cmake] [100%] Linking CXX executable Tutorial [cmake_step2:cmake] [100%] Built target Tutorial

The resulting executable binary can be found at var/build/x86_64/Tutorial. Let's declare it a build artifact by mentioning it a new artifacts file with the following content.

Tutorial

To run the program, we need a runtime package that is slightly more advanced than the first hello example. This time, we need declare that our runtime requires content from other depot archives in addition to our program by creating a file pkg/cmake_step2/archives with the following content:

nfeske/src/posix nfeske/src/libc nfeske/src/vfs nfeske/src/stdcxx

This way, our sub system incorporates the shared libraries found in those depot archives. A suitable pkg/cmake_step2/runtime for running the program within a Genode scenario looks like this:

<runtime ram="10M" caps="1000" binary="Tutorial"> <config> <libc stdout="/dev/log" stderr="/dev/log"/> <vfs> <dir name="dev"> <log/> </dir> </vfs> <arg value="Tutorial"/> <arg value="24"/> </config> <content> <rom label="Tutorial"/> <rom label="posix.lib.so"/> <rom label="libc.lib.so"/> <rom label="libm.lib.so"/> <rom label="stdcxx.lib.so"/> <rom label="vfs.lib.so"/> </content> </runtime>

Since the tutorial uses the C runtime, we have to supply a configuration that defines how the virtual file system of the component looks like, and where the program's standard output should go. We also specify the first and second arguments of the POSIX program as "Tutorial" (name of the program) and "24" its actual argument. The <content> lists all ROM modules required.

With this runtime package in place, let's give the Tutorial a run:

cmake_step2/$ goa run [cmake_step2:cmake] -- Configuring done [cmake_step2:cmake] -- Generating done [cmake_step2:cmake] -- Build files have been written to: .../cmake_step2/var/build/x86_64 [cmake_step2:cmake] [100%] Built target Tutorial download nfeske/bin/x86_64/base-linux/2020-02-27.tar.xz download nfeske/bin/x86_64/base-linux/2020-02-27.tar.xz.sig download nfeske/bin/x86_64/init/2020-02-27.tar.xz download nfeske/bin/x86_64/init/2020-02-27.tar.xz.sig download nfeske/bin/x86_64/libc/2020-02-27.tar.xz download nfeske/bin/x86_64/libc/2020-02-27.tar.xz.sig download nfeske/bin/x86_64/posix/2020-02-27.tar.xz download nfeske/bin/x86_64/posix/2020-02-27.tar.xz.sig download nfeske/bin/x86_64/stdcxx/2020-02-19.tar.xz download nfeske/bin/x86_64/stdcxx/2020-02-19.tar.xz.sig download nfeske/bin/x86_64/vfs/2020-02-27.tar.xz download nfeske/bin/x86_64/vfs/2020-02-27.tar.xz.sig download nfeske/src/base-linux/2020-02-27.tar.xz download nfeske/src/base-linux/2020-02-27.tar.xz.sig download nfeske/src/init/2020-02-27.tar.xz download nfeske/src/init/2020-02-27.tar.xz.sig download nfeske/src/libc/2020-02-27.tar.xz download nfeske/src/libc/2020-02-27.tar.xz.sig download nfeske/src/posix/2020-02-27.tar.xz download nfeske/src/posix/2020-02-27.tar.xz.sig download nfeske/src/stdcxx/2020-02-19.tar.xz download nfeske/src/stdcxx/2020-02-19.tar.xz.sig download nfeske/src/vfs/2020-02-27.tar.xz download nfeske/src/vfs/2020-02-27.tar.xz.sig download nfeske/api/block_session/2020-02-27.tar.xz download nfeske/api/block_session/2020-02-27.tar.xz.sig download nfeske/api/file_system_session/2019-11-18.tar.xz download nfeske/api/file_system_session/2019-11-18.tar.xz.sig download nfeske/api/os/2020-02-27.tar.xz download nfeske/api/os/2020-02-27.tar.xz.sig download nfeske/api/report_session/2019-02-25.tar.xz download nfeske/api/report_session/2019-02-25.tar.xz.sig download nfeske/api/rtc_session/2019-08-20.tar.xz download nfeske/api/rtc_session/2019-08-20.tar.xz.sig download nfeske/api/so/2019-02-25.tar.xz download nfeske/api/so/2019-02-25.tar.xz.sig download nfeske/api/terminal_session/2019-02-25.tar.xz download nfeske/api/terminal_session/2019-02-25.tar.xz.sig download nfeske/api/timer_session/2019-11-25.tar.xz download nfeske/api/timer_session/2019-11-25.tar.xz.sig download nfeske/api/vfs/2020-02-27.tar.xz download nfeske/api/vfs/2020-02-27.tar.xz.sig Genode 20.02-1-gac1b2ec24e 17592186044415 MiB RAM and 8997 caps assigned to init [init -> cmake_step2] The square root of 24 is 4.89898

We see that Goa takes care of downloading all dependencies needed to host the subsystem and subsequently executes the scenario. The program built by the tutorial prints the result "The square root of 24 is 4.89898".

The next step

The intro above covers just a tiny part of the scope of Goa.

For further exploration, read the next episode about using Goa to create a little Unix...

Please also feel welcome to explore Goa on your own. A good starting point would be the built-in help command:

goa help

If you have feedback, please don't hesitate to reach out!

Edit (2020-03-02): updated to Genode 20.02