Hello world! I'm excited to announce the first release of MUnit, a new Scala testing library with a focus on actionable errors and extensible APIs. You may be thinking "Why create Yet Another Scala testing library?". It's a good question and this post is my attempt to explain the motivations for creating MUnit.

Like many other existing testing libraries, MUnit has no external Scala dependencies and is published for a wide range of compiler versions and platforms.

Scala Version JVM Scala.js (0.6.x, 1.x) Native (0.4.x) 2.11.x ✅ ✅ ✅ 2.12.x ✅ ✅ n/a 2.13.x ✅ ✅ n/a 0.21.x (Dotty) ✅ n/a n/a

MUnit tries to distinguish itself by focusing on the following features:

Tests as values : test cases are represented as normal data structures that you can manipulate and abstract over.

: test cases are represented as normal data structures that you can manipulate and abstract over. Rich filtering capabilities : MUnit provides fine-grained control over what tests are enabled for which environments.

: MUnit provides fine-grained control over what tests are enabled for which environments. Actionable errors : the formatting of failed test cases is optimized for giving you as much information as possible to understand how to fix the test case.

: the formatting of failed test cases is optimized for giving you as much information as possible to understand how to fix the test case. Tooling integrations : MUnit is implemented as a JUnit runner and tries to build on top of existing JUnit functionality where possible.

: MUnit is implemented as a JUnit runner and tries to build on top of existing JUnit functionality where possible. Insightful test reports: the MUnit sbt plugin allows you to analyze historical data about your tests to answer questions like "is this test suite flaky?" and "which tests are slowing down my CI?".

To use MUnit, first add a dependency in your build.

libraryDependencies += "org.scalameta" %% "munit" % "0.4.3" testFrameworks += new TestFramework ( "munit.Framework" )

Next, write a test case:

class MySuite extends munit . FunSuite { test( "hello" ) { assert( 41 == 42 ) } }

Check out the getting started guide.

Tests as values

If you know how to write normal Scala programs you should feel comfortable reasoning about how MUnit works.

Internally, a core MUnit data structure is GenericTest[T] , which represents a single test case and is roughly defined like this.

case class GenericTest [ T ]( name: String , body: ( ) => T , tags: Set [ Tag ], location: Location ) abstract class Suite { type TestValue type Test = GenericTest [ TestValue ] def munitTests (): Seq [ Test ] }

A test suite returns a Seq[Test] , which you as a user can generate and abstract over any way you like.

Importantly, MUnit test cases are not discovered via runtime reflection like in JUnit and MUnit test cases are not generated via macros like in utest.

MUnit provides a high-level API to write tests in a ScalaTest-inspired FunSuite syntax where the type parameter for GenericTest[T] is defined as Future[Any] .

abstract class FunSuite extends Suite { type TestValue = Future [ Any ] }

For common usage of MUnit you are not expected to write raw GenericTest[T](...) expressions but knowing this underlying data model helps you implement features like test retries, disabling tests based on dynamic conditions, enforce stricter type safety and more.

Rich filtering capabilities

Using tags, MUnit provides a extensible way to disable/enable tests based on static and dynamic conditions.

For example, the MUnit codebase itself is cross-built against 11 different combinations of Scala compiler versions (2.11, 2.12, 2.13, Dotty) and platforms (JVM,JS,Native). Our CI also runs tests on JDK 8/11 and Linux/Windows. Inevitably, some test cases end up getting disabled in certain environments.

Imagine that we have test case that for some reason should only run on Windows in Scala 2.13. We can implement a custom Window213 tag with the following code:

import scala.util. Properties import munit._ object Windows213 extends Tag ( " Windows213 " ) class MySuite extends FunSuite { override def munitTestTransforms = super .munitTestTransforms ++ List ( new TestTransform ( "Windows213" , { test => val isIgnored = test.tags( Windows213 ) && !( Properties .isWin && Properties .versionNumberString.startsWith( "2.13" ) ) if (isIgnored) test.tag( Ignore ) else test }) ) test( "windows-213" .tag( Windows213 )) { } test( "normal test" ) { } }

By encoding the environment requirements in the test implementation, we prevent the situation where users run sbt test commands that are invalid for their active operating system or Scala version.

Check out the filtering tests guide to learn more how to enable/disable tests with MUnit.

Actionable errors

The design goal for MUnit error messages is to give you as much context as possible to address the test failure. Let's consider a few concrete examples.

In the image above, you can cmd+click on the .../test/scala/munit/DemoSuite.scala:7 path to open the failing line of code in your editor. By highlighting the failing line of code, you also immediately gain some understanding for why the test might be failing.

In the image above, the failing assertEquals() displays a diff comparing two values of a User case class. The "Obtained" section includes copy-paste friendly syntax of the obtained value, which can be helpful in the common situation when a failing test case should have passed because the expected behavior of your program has changed.

In the image above, the failing assertNoDiff() includes a stripMargin formatted multiline string of the obtained string. The assertNoDiff() assertions is helpful for comparing multiline strings ignoring non-visible differences such as Windows/Unix newlines, ANSI color codes and leading/trailing whitespace.

In the image above, the clue(a) helpers are used to enrich the error message with additional information that is displayed when the assertion fails.

In the image above, stack trace elements that are defined from library dependencies like the standard library are grayed out making it easier to find stack trace elements that are relevant for your code. This can be helpful when debugging large exception stack traces. This feature is inspired by the pretty-printing of stack traces in utest.

Check out the writing assertions guide to learn more how to write assertions with helpful error messages.

Tooling integrations

The tooling side of a testing library is equally important as the library APIs. MUnit is implemented as a JUnit runner, which means that any existing tool that knows how to run a JUnit test suite knows how to run MUnit test suites.

For example, IntelliJ already detects MUnit test suites even if IntelliJ has no custom logic to support MUnit.

Likewise, build tools such as Gradle and Pants can integrate with MUnit using their existing JUnit integrations.

Insightful test reports

MUnit has an sbt plugin to store structured JSON data about test results in Google Cloud. The data can then be used to generate HTML reports based on historical test data.

The image below shows test cases in the Metals codebase sorted by how frequently they fail on the master branch.

Click on image to open full report

The Metals test suite ignores failures in tests that are tagged as flaky. However, it's clear that DefinitionLspSuite.missing-compiler-plugin is not flaky, it consistently fails on every run. On the other hand, PantsLspSuite.basic has only failed once out of eleven test runs so it seems to be legitimately flaky.

The Metals codebase has ~1.5k test cases, some which run against up to seven different Scala compiler versions. It's not ideal that some test cases fail non-deterministically but it's normal that it happens as the project grows and we support more build tools, Scala versions and features. While there is no silver bullet for avoiding flaky test failures, having data about how frequently a test fails is at least a starting point to begin addressing the problem.

Check out the generating test reports guide to learn how to configure your build to upload test reports to Google Cloud using the MUnit sbt plugin. The plugin is implemented as an sbt TestsListener so should work with any testing library (including ScalaTest, utest, ...) although it has so far only been tested against MUnit.

Credits

I want to thank @gabro for implementing Dotty support, porting the Metals codebase to MUnit and sharing tons of valuable feedback. Without your initial interest in MUnit I probably would not have polished the project for a proper release.

Conclusion

MUnit is a new Scala testing library with a focus on actionable errors and extensible APIs. MUnit is already used in several Scalameta projects including scalameta/scalameta, scalameta/metals and scalameta/mdoc.

Most of the ideas in this post are not new. The features in MUnit are heavily inspired by existing testing libraries including ScalaTest, utest, JUnit and ava (a JavaScript testing library). However, I'm not aware of a testing library that provides the combination of all the features presented in this post in one solution and I hope that explains the motivation for why MUnit exists.

Happy testing ✌️