You have probably encountered this problem while working with SBT and bigger projects. I’m talking about compilation times and test execution times, in other words, having to wait instead of working. Imagine working with a build tool that rebuilds only what is necessary, using a distributed cache, so if module A is built by one of your team members you won’t have to do it again. Or imagine being able to run builds of different parts of your project in parallel, or run tests only for the affected code that has been changed. Sounds promising right? That’s why, in this tutorial, I will be showing you what Bazel build is and how to set your project up in Scala.

Introduction to Bazel build

Bazel is a build tool from Google, which allows you to easily manage builds and tests in huge projects. This tool gives huge flexibility when it comes to the configuration and granularity of the basic build unit. It can be a set of packages, one package or even just one file. The basic build unit is called a target, the target is an instance of rules. A rule is a function that has a set of inputs and outputs; if the inputs do not change then the outputs stay the same. By having more targets (the disadvantage of this solution is having more build files to maintain) where not all of them depend on each other, more builds can run in parallel because Bazel build uses incremental builds, so it rebuilds only the part of the dependency graph that has been changed, as well as only running tests for the affected parts.

It can distribute, build and test actions across multiple machines, and then build and reuse previously done cached work, which makes your builds even more scalable.

Bazel can also print out a dependency graph, the results of which can be visualized on this page

So if your project takes a lot of time to build, and you don’t want to waste any more time, this tool is what you need.

Speed up your compile times, speed up your tests, speed up your whole team’s work!

In this tutorial, we will be using Bazel version 1.0.0.

Project structure

We will be working on a project with this structure:

├── BUILD

├── WORKSPACE

├── bazeltest

│ ├── BUILD

│ └── src

│ ├── main

│ │ └── scala

│ │ └── bazeltest

│ │ └── Main.scala

│ └── test

│ └── scala

│ └── bazeltest

│ └── MainSpec.scala

├── dependencies.yaml

└── othermodule

├── BUILD

└── src

├── main

│ └── scala

│ └── othermodule

│ └── Worker.scala

└── test

└── scala

└── othermodule

└── WorkerSpec.scala

So we have two modules called: bazeltest and othermodule.

Bazeltest will depend on othermodule.

Workspace file setup

Each project has one WORKSPACE file, where we will define things like Scala version and dependencies. If in the project directory there is a subdirectory with a WORKSPACE file, then while doing our builds this subdirectory will be omitted.

To make it work with Scala, then let’s take an already prepared boilerplate WORKSPACE file from:

https://github.com/bazelbuild/rules_scala#getting-started

Be aware of the change in rules_scala_version. Rules_scala_version is commit’s sha. So if you want to use the newest version of the rules, check GitHub repository and copy-paste commit’s sha.

We also have to add at the end of the file:

load(“//3rdparty:workspace.bzl”, “maven_dependencies”) maven_dependencies()

This will be used by a third-party tool called bazel-deps, but we will come back to this at the next step.

So after the changes:

rules_scala_version=“0f89c210ade8f4320017daf718a61de3c1ac4773” # update this as needed load(“@bazel_tools//tools/build_defs/repo:http.bzl”, “http_archive”) http_archive( name = “io_bazel_rules_scala”, strip_prefix = “rules_scala-%s” % rules_scala_version, type = “zip”, url = “https://github.com/bazelbuild/rules_scala/archive/%s.zip” % rules_scala_version, ) load(“@io_bazel_rules_scala//scala:toolchains.bzl”, “scala_register_toolchains”) scala_register_toolchains() load(“@io_bazel_rules_scala//scala:scala.bzl”, “scala_repositories”) scala_repositories() # bazel-skylib 0.8.0 released 2019.03.20 (https://github.com/bazelbuild/bazel-skylib/releases/tag/0.8.0) skylib_version = “0.8.0” http_archive( name = “bazel_skylib”, type = “tar.gz”, url = “https://github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib.{}.tar.gz”.format (skylib_version, skylib_version), sha256 = “2ef429f5d7ce7111263289644d233707dba35e39696377ebab8b0bc701f7818e”, ) load(“//3rdparty:workspace.bzl”, “maven_dependencies”) maven_dependencies() scala_repositories(( “2.12.8”, { “scala_compiler”: “f34e9119f45abd41e85b9e121ba19dd9288b3b4af7f7047e86dc70236708d170”, “scala_library”: “321fb55685635c931eba4bc0d7668349da3f2c09aee2de93a70566066ff25c28”, “scala_reflect”: “4d6405395c4599ce04cea08ba082339e3e42135de9aae2923c9f5367e957315a” } ))

If you wish to set a specific Scala version, add code from:

https://github.com/bazelbuild/rules_scala#getting-started

scala_repositories(( "2.12.8", { "scala_compiler": "f34e9119f45abd41e85b9e121ba19dd9288b3b4af7f7047e86dc70236708d170", "scala_library": "321fb55685635c931eba4bc0d7668349da3f2c09aee2de93a70566066ff25c28", "scala_reflect": "4d6405395c4599ce04cea08ba082339e3e42135de9aae2923c9f5367e957315a" } ))

In this file, we will setup the Scala rules and everything else that is needed to compile the Scala project.

BUILD files setup

To write BUILD files we will use the following methods:

load – which loads the Bazel Scala rules, and extensions scala_binary – generates a Scala executable scala_library – generates a .jar file from Scala source files. scala_test – generates a Scala executable that runs unit test suites written using the scalatest library.

Start from the BUILD file in a project folder.

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_binary") scala_binary( name = "App", deps = [ "//bazeltest" ], main_class = "bazeltest.Main" )

We have named it App, just one dependency to the bazeltest package.

In deps, we list our dependencies, where our own modules or third party can be. Main_class is our entry point.

In the bazeltest package BUILD file:

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test") scala_library( name = "bazeltest", srcs = ["src/main/scala/bazeltest/Main.scala"], deps = [ "//othermodule", "//3rdparty/jvm/joda_time:joda_time" ], visibility = ["//:__pkg__"] ) scala_test( name = "test-main", srcs = ["src/test/scala/bazeltest/MainSpec.scala"], deps = [":bazeltest"] )

Our Main.scala file will use some external third party dependency such as joda date time, and Worker from the subpack package.

In srcs we set our Main.scala file, but it could be a list of files, listed one by one or a matching path pattern for example:

glob(["src/main/scala/bazeltest/*.scala"])

glob(["src/main/scala/bazeltest/**/*..scala"])

and in deps all the necessary dependencies, so for this example our own sub pack package plus third part joda date time. For now, it points to the 3rdparty folder which does not exist yet, this will be done at one of the next steps so don’t worry.

Visibility is used to define which other targets can use this target as a dependency, in this example, we define a project folder containing the main BUILD file.

Now the BUILD file for othermodule:

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test") scala_library( name = "othermodule", srcs = glob(["src/main/scala/othermodule/*.scala"]), deps = [], visibility = ["//bazeltest:__pkg__"] ) scala_test( name = "test-othermodule", srcs = ["src/test/scala/othermodule/WorkerSpec.scala"], deps = [":othermodule"] )

Here we have set up a visibility param to the bazeltest package. So only this package can read from this one. If other packages try to reach this, we will see an error.

Dependencies

We will use a third-party tool for this:

https://github.com/johnynek/bazel-deps

Open the dependencies.yaml file and put this there:

options: buildHeader: [ "load(\"@io_bazel_rules_scala//scala:scala_import.bzl\", \"scala_import\")", "load(\"@io_bazel_rules_scala//scala:scala.bzl\", \"scala_library\", \"scala_binary\", \"scala_test\")" ] languages: [ "java", "scala:2.12.8" ] resolverType: "coursier" resolvers: - id: "mavencentral" type: "default" url: https://repo.maven.apache.org/maven2/ - id: "hmrc" type: "default" url: https://hmrc.bintray.com/releases strictVisibility: true transitivity: runtime_deps versionConflictPolicy: highest dependencies: joda-time: joda-time: lang: java version: "2.10.4" com.typesafe.scala-logging: scala-logging: lang: scala version: "3.9.0" com.typesafe.akka: akka-http: lang: scala version: "10.1.7" org.scalatest: scalatest: lang: scala version: "3.0.8" replacements: org.scala-lang: scala-library: lang: scala/unmangled target: "@io_bazel_rules_scala_scala_library//:io_bazel_rules_scala_scala_library" scala-reflect: lang: scala/unmangled target: "@io_bazel_rules_scala_scala_reflect//:io_bazel_rules_scala_scala_reflect" scala-compiler: lang: scala/unmangled target: "@io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler" org.scala-lang.modules: scala-parser-combinators: lang: scala target: "@io_bazel_rules_scala_scala_parser_combinators//:io_bazel_rules_scala_scala_parser_combinators" scala-xml: lang: scala target: “@io_bazel_rules_scala_scala_xml//:io_bazel_rules_scala_scala_xml"

(Language is always required and may be one of java, Scala, Scala/unmangled.

This is important, if you define an invalid language then errors will occur.

Replacements are used for internal targets instead of Maven ones.)

Save the system variable of this project path, for example (working on a Mac):

export MY_PROJ_DIR=`pwd`

We will need this in a minute.

Clone

https://github.com/johnynek/bazel-deps

and enter the bazel-deps folder.

Ensure that this tool uses the same rules_scala commit sha.

Open the WORKSPACE file inside the bazel-deps and look for this:

git_repository( name = "io_bazel_rules_scala", remote = "https://github.com/bazelbuild/rules_scala", commit = "0f89c210ade8f4320017daf718a61de3c1ac4773" # HEAD as of 2019-10-17, update this as needed )

Commit is of course what we need to change ( if it is different than in our WORKSPACE file in rules_scala_version ).

bazel run //:parse generate -- --repo-root "$MY_PROJ_DIR" --sha-file 3rdparty/workspace.bzl

--deps dependencies.yaml

This will download dependencies into a 3rdparty folder into your project directory.

INFO: Analyzed target //:parse (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //src/scala/com/github/johnynek/bazel_deps:parseproject up-to-date: bazel-bin/src/scala/com/github/johnynek/bazel_deps/parseproject bazel-bin/src/scala/com/github/johnynek/bazel_deps/parseproject.jar INFO: Elapsed time: 0.168s, Critical Path: 0.01s INFO: 0 processes. INFO: Build completed successfully, 1 total action INFO: Build completed successfully, 1 total action wrote 26 targets in 8 BUILD files

The first run

Before doing the first run, let’s implement our Main and Worker classes.

package bazeltest import othermodule.Worker import org.joda.time.DateTime object Main extends App { println("IN MAIN now: "+DateTime.now().plusYears(11)) val worker = new Worker worker.doSomething() def status(): String = "OKi" }

package othermodule class Worker { def doSomething() : Int = { println("Doing something") 12345 } def pureFunc(): String = "ABC" }

bazel run //:App

INFO: Analyzed target //:App (1 packages loaded, 2 targets configured). INFO: Found 1 target... INFO: From Linking external/com_google_protobuf/libprotobuf_lite.a [for host]: /Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/io_win32.o has no symbols INFO: From Linking external/com_google_protobuf/libprotobuf.a [for host]: /Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/error_listener.o has no symbols INFO: From Building external/com_google_protobuf/libprotobuf_java.jar (122 source files, 1 source jar): warning: -parameters is not supported for target value 1.7. Use 1.8 or later. Target //:App up-to-date: bazel-bin/App bazel-bin/App.jar INFO: Elapsed time: 52.246s, Critical Path: 23.22s INFO: 194 processes: 189 darwin-sandbox, 5 worker. INFO: Build completed successfully, 198 total actions INFO: Build completed successfully, 198 total actions IN MAIN now: 2030-10-11T11:26:07.533+01:00 Doing something

The first run takes some time because it has to download the dependencies, so don’t worry.

Unit tests

Now let’s write some simple unit tests:

package bazeltest import org.scalatest._ class MainSpec extends FlatSpec with Matchers { "status" should "return OK" in { Main.status() shouldBe "OKi" } }

package othermodule import org.scalatest._ class WorkerSpec extends FlatSpec with Matchers { val worker = new Worker() "do something" should "return 12345" in { worker.doSomething() shouldBe 12345 } "pureFunc" should "return ABC" in { worker.pureFunc() shouldBe "ABC"

And run them:

bazel test //bazeltest:test-main

INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured). INFO: Found 1 test target... Target //bazeltest:test-main up-to-date: bazel-bin/bazeltest/test-main bazel-bin/bazeltest/test-main.jar INFO: Elapsed time: 1.047s, Critical Path: 0.89s INFO: 3 processes: 2 darwin-sandbox, 1 worker. INFO: Build completed successfully, 4 total actions //bazeltest:test-main PASSED in 0.5s Executed 1 out of 1 test: 1 test passes. INFO: Build completed successfully, 4 total actions

bazel test //othermodule:test-othermodule

INFO: Analyzed target //othermodule:test-othermodule (0 packages loaded, 0 targets configured). INFO: Found 1 test target... Target //othermodule:test-othermodule up-to-date: bazel-bin/othermodule/test-othermodule bazel-bin/othermodule/test-othermodule.jar INFO: Elapsed time: 1.438s, Critical Path: 1.29s INFO: 2 processes: 1 darwin-sandbox, 1 worker. INFO: Build completed successfully, 3 total actions //othermodule:test-othermodule PASSED in 0.6s Executed 1 out of 1 test: 1 test passes. INFO: Build completed successfully, 3 total actions

Try now to change the status method from Main, to return “OK” instead of “OKi”.

Run the tests again:

bazel test //bazeltest:test-main

INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured). INFO: Found 1 test target... FAIL: //bazeltest:test-main (see /private/var/tmp/_bazel_maciejbak/16727409c9f0575889b09993f53ce424/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/bazeltest/test-main/test.log) Target //bazeltest:test-main up-to-date: bazel-bin/bazeltest/test-main bazel-bin/bazeltest/test-main.jar INFO: Elapsed time: 1.114s, Critical Path: 0.96s INFO: 3 processes: 2 darwin-sandbox, 1 worker. INFO: Build completed, 1 test FAILED, 4 total actions //bazeltest:test-main FAILED in 0.6s /private/var/tmp/_bazel_maciejbak/16727409c9f0575889b09993f53ce424/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/bazeltest/test-main/test.log INFO: Build completed, 1 test FAILED, 4 total actions

bazel test //othermodule:test-othermodule

INFO: Analyzed target //othermodule:test-othermodule (0 packages loaded, 0 targets configured). INFO: Found 1 test target... Target //othermodule:test-othermodule up-to-date: bazel-bin/othermodule/test-othermodule bazel-bin/othermodule/test-othermodule.jar INFO: Elapsed time: 0.150s, Critical Path: 0.00s INFO: 0 processes. INFO: Build completed successfully, 1 total action //othermodule:test-othermodule (cached) PASSED in 0.6s Executed 0 out of 1 test: 1 test passes. INFO: Build completed successfully, 1 total action

Bazel build sees what has been changed, and runs tests only for the affected classes.

So test results for othermodule are taken from the cache, and only the main tests run.

The test failed because we didn’t change the results in the Spec file, so the change expected the result in the test to the Main.status() shouldBe “OK”.

Run tests again:

bazel test //bazeltest:test-main

INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured). INFO: Found 1 test target... Target //bazeltest:test-main up-to-date: bazel-bin/bazeltest/test-main bazel-bin/bazeltest/test-main.jar INFO: Elapsed time: 1.389s, Critical Path: 1.22s INFO: 2 processes: 1 darwin-sandbox, 1 worker. INFO: Build completed successfully, 3 total actions //bazeltest:test-main PASSED in 0.6s Executed 1 out of 1 test: 1 test passes. INFO: Build completed successfully, 3 total actions

Dependency graph

We can easily visualize our dependency graph:

In the command line run:

bazel query --noimplicit_deps "deps(//:App)" --output graph

digraph mygraph { node [shape=box]; "//:App" "//:App" -> "//bazeltest:bazeltest" "//bazeltest:bazeltest" "//bazeltest:bazeltest" -> "//bazeltest:src/main/scala/bazeltest/Main.scala" "//bazeltest:bazeltest" -> "//3rdparty/jvm/joda_time:joda_time" "//bazeltest:bazeltest" -> "//othermodule:othermodule" "//othermodule:othermodule" "//othermodule:othermodule" -> "//othermodule:src/main/scala/othermodule/Worker.scala" "//othermodule:src/main/scala/othermodule/Worker.scala" "//3rdparty/jvm/joda_time:joda_time" "//3rdparty/jvm/joda_time:joda_time" -> "//external:jar/joda_time/joda_time" "//external:jar/joda_time/joda_time" "//external:jar/joda_time/joda_time" -> "@joda_time_joda_time//jar:jar" "//bazeltest:src/main/scala/bazeltest/Main.scala" "@joda_time_joda_time//jar:jar" "@joda_time_joda_time//jar:jar" -> "@joda_time_joda_time//jar:joda_time_joda_time.jar

@joda_time_joda_time//jar:joda_time_joda_time-sources.jar" "@joda_time_joda_time//jar:joda_time_joda_time.jar

@joda_time_joda_time//jar:joda_time_joda_time-sources.jar" } Loading: 12 packages loaded

Paste results to

Generate jar

bazel build //:App

INFO: Analyzed target //:App (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //:App up-to-date: bazel-bin/App bazel-bin/App.jar INFO: Elapsed time: 0.085s, Critical Path: 0.00s INFO: 0 processes. INFO: Build completed successfully, 1 total action

Bazel build: Summary

( then we use glob ), could even be a package with all the subpackages, such as:

In this post, we showed what is bazel, when to use it, and how to make basic configuration. It can take some time to properly set up complex projects using bazel build, but I guarantee you, in the end, it will speed up the whole team’s work.

Useful links