ScalaTest 2.0 Release Notes

ScalaTest 2.0 is a major upgrade to ScalaTest, the culmination of over two years of effort. Although it includes many enhancements, we kept a close eye on compatibility with existing ScalaTest 1.x code. We also tried hard to preserve what people liked about ScalaTest 1.x: that it is simple, the code is clean and clear, it's fully documented, and because of very thorough testing, it “just works.”

For information on how to include ScalaTest in your project, see the download page.

Highlights

ScalaTest 2.0 is such a big upgrade that you could easily get lost all in the details of what's new. This section gives the main highlights subdivided into general categories, with links to more detail. To get a quick idea of what's new in ScalaTest 2.0, scan through this section.

Equality enhancements

Suite enhancements

Added an Outcome result type to withFixture .

result type to . Added a Status result type to Suite 's run methods, and used Status to ensure BeforeAndAfterAll 's afterAll method is executed after any parallel execution completes.

result type to 's methods, and used to ensure 's method is executed after any parallel execution completes. Refactored Suite 's run methods to take an Args , to make overriding nicer.

's methods to take an , to make overriding nicer. Added markup to style traits to include markup text for specifications.

to style traits to include markup text for specifications. Added note and alert to stye traits for sending status notifications from tests.

and to stye traits for sending status notifications from tests. Added Sequential and Stepwise (to pre-existing Suites ) combinators for suite composition.

and (to pre-existing ) combinators for suite composition. Added support for they (in addition to it ) to FunSpec and FlatSpec .

(in addition to ) to and . Added ability to tag all tests in a suite class by annotating the class with a tag annotation. For example, can ignore all tests in a suite class with @Ignore .

. Added DynaTags and a dynaTags field to the primary constructor of class Filter to enable the dynamic tagging of tests. Dynamic tagging facilitates features such as test name wildcards and rerunning previously failed tests; It enables selection of tests even in nested suites that can't be instantiated directly.

and a field to the primary constructor of class to enable the dynamic tagging of tests. Dynamic tagging facilitates features such as test name wildcards and rerunning previously failed tests; It enables selection of tests even in nested suites that can't be instantiated directly. Added the suiteId lifecycle method to trait Suite . This new lifecycle method is intended to provide a unique ID for each suite that executes in a suite of tests, so that tests can be dynamically tagged. (A dynamic tag identifies tests by suiteId and testName .)

lifecycle method to trait . This new lifecycle method is intended to provide a unique ID for each suite that executes in a suite of tests, so that tests can be dynamically tagged. (A dynamic tag identifies tests by and .) Added a rerunner lifecycle method to Suite , which provides an optional, fully qualified name of a suite (with a public, no-arg constructor) that can be used create to rerun its tests. If class has a public, no-arg constructor, then that class's fully qualified name can be returned from rerunner . If it does not have a public, no-arg constructor, but rather is created as a nested suite of a class that does have a public, no-arg constructor, then the nested class can return from rerunner the fully qualified name of that outer, nesting class. To rerun a test in the nested class, ScalaTest can create the nesting class via its public, no-arg constructor, and run it. The nesting class will create the actual class in which the test resides, and run it as a nested suite, thereby rerunning the test.

lifecycle method to , which provides an optional, fully qualified name of a suite (with a public, no-arg constructor) that can be used create to rerun its tests. If class has a public, no-arg constructor, then that class's fully qualified name can be returned from . If it does not have a public, no-arg constructor, but rather is created as a nested suite of a class that does have a public, no-arg constructor, then the nested class can return from the fully qualified name of that outer, nesting class. To rerun a test in the nested class, ScalaTest can create the nesting class via its public, no-arg constructor, and run it. The nesting class will create the actual class in which the test resides, and run it as a nested suite, thereby rerunning the test. Added a boolean excludeNestedSuites field to the primary constructor of class Filter . This field supports rerunning selected (such as previously failed) tests.

field to the primary constructor of class . This field supports rerunning selected (such as previously failed) tests. Added traits Spec and fixture.Spec to serve as the new style trait in which tests are methods, and deprecated the use of Suite as a style trait.

and to serve as the new style trait in which tests are methods, and deprecated the use of as a style trait. Added a @DoNotDiscover annotation that prevents discovery of an otherwise discoverable Suite class.

SuiteMixin enhancements

Added trait SuiteMixin to serve as the new base class for stackable traits that can be mixed into Suite , and deprecated trait AbstractSuite .

to serve as the new base class for stackable traits that can be mixed into , and deprecated trait . Added TestData trait, made NoArgTest and OneArgTest implement it, and added trait BeforeAndAfterEachTestData to faciliate using the test name, text, scopes, and tags, as well as the config map, in withFixture , beforeEach , and afterEach methods.

trait, made and implement it, and added trait to faciliate using the test name, text, scopes, and tags, as well as the config map, in , , and methods. Added BeforeAndAfterAllConfigMap to faciliate using the config map in beforeAll and afterAll methods.

to faciliate using the config map in and methods. Changed the behavior of BeforeAndAfterAll so that it only calls beforeAll and afterAll in the outer instance of a OneInstancePerTest or ParallelTestExecution .

so that it only calls and in the outer instance of a or . Enhanced BeforeAndAfterAll and BeforeAndAfterAllConfigMap to take expectedTestCount into consideration when deciding whether to invoke beforeAll and afterAll .

and to take into consideration when deciding whether to invoke and . Modified OneInstancePerTest so that it calls super.runTests which emits events for scopes (such as describe clauses in FunSpec ).

so that it calls which emits events for scopes (such as clauses in ). Similarly, modified ParallelTestExecution so that it calls super.runTests which emits events for scopes (such as describe clauses in FunSpec ).

so that it calls which emits events for scopes (such as clauses in ). Added a Retries trait (and related Retryable tag) and a Catcher class to help deal with flickers: tests that usually pass but occasionally fail.

Assertions enhancements

Added the ability to cancel a test. Added assume and cancel methods to Assertions , and added a cancelAfter method to Timeouts .

and methods to , and added a method to . Reimplemented assert and assume as macros that produce descriptive error messages in Assertions .

and as macros that produce descriptive error messages in . Added Inspectors ( forAll , forEvery , forAtLeast , forAtMost , forBetween , forExactly ) to enable assertions about collections, including nested collections.

( , , , , , ) to enable assertions about collections, including nested collections. Added LoneElement to assert against single-element collections.

to assert against single-element collections. Added TryValues to facilitate assertions about Try values.

to facilitate assertions about values. Added AsyncAssertions to org.scalatest.concurrent .

to . Added ScalaFutures to help write tests involving Scala futures.

to help write tests involving Scala futures. Added Checkpoints , which allows you to accumulate and report multiple assertions in a single test.

Matchers enhancements

Runner enhancements

For a table of all the available options, see the main Scaladoc documentation for Runner .

. Added support for reminders, a summary of failed and/or canceled tests that can be optionally printed out at the end of a run in string (stdout, stderr, file) reporters. This obviates the need to scroll backwards in search of details about failed and/or canceled tests in console output, and also ensures that such details won't have scrolled off the top and out of existence.

Enhanced -s to take a glob in addition to a complete, fully-qualified suite class name.

to take a glob in addition to a complete, fully-qualified suite class name. Added slowpoke notifications, which emit AlertProvided events periodically listing tests that have been running longer than a specified time limit. (-W)

events periodically listing tests that have been running longer than a specified time limit. (-W) Added ability to memorize and rerun failed and canceled tests. (-M and -A)

Added to Runner a way to specify tests to run by full or partial ("wildcard") name ( -z ).

a way to specify tests to run by full or partial ("wildcard") name ( ). Enabled on-the-fly sorting of events (with a timeout) during parallel runs ( -PS ).

). Added unformatted mode to stdout (-oU), stderr ( -eU ), and file reporters ( -fU ) for helping debug problematic parallel test runs.

), and file reporters ( ) for helping debug problematic parallel test runs. Added a socket reporter (-k), which serializes events to a socket. This facilitates running tests remotely, such as with sbt's forking feature.

Added a -i command line argument to Runner to provide a way to specify a suite by suite ID. This complements the -s command, which allows you to identify a suite by fully qualified name. The -i command was added primarily to allow IDEs and other tools to rerun failed (or otherwise selected) tests.

command line argument to to provide a way to specify a suite by suite ID. This complements the command, which allows you to identify a suite by fully qualified name. The command was added primarily to allow IDEs and other tools to rerun failed (or otherwise selected) tests. Added a -t to Runner , a way to specify a specific test name. A -t clauses may follow -s or -i , or be standalone. All -t 's immediately following a -s or -i will be considered tests of the suite specified by that -s or -i . If -t <test name> appears standalone, Runner will execute any tests with matching names on any suite discovered in the runpath.

to , a way to specify a specific test name. A clauses may follow or , or be standalone. All 's immediately following a or will be considered tests of the suite specified by that or . If appears standalone, will execute any tests with matching names on any suite discovered in the runpath. Added a -z argument to Runner , a means to specify test names via a wildcard: any test whose string name includes the string specified with -z will be selected. -z clauses may follow -s or -i clauses, or be standalone. All -z 's immediately following a -s or -i will be considered wildcards selecting tests of the suite by that -s or -i . If -z <test name wildcard> appears standalone, Runner will execute any tests with matching names on any suite discovered in the runpath.

argument to , a means to specify test names via a wildcard: any test whose string name includes the string specified with will be selected. clauses may follow or clauses, or be standalone. All 's immediately following a or will be considered wildcards selecting tests of the suite by that or . If appears standalone, will execute any tests with matching names on any suite discovered in the runpath. Added the ability to sort events on-the-fly when suites are run in parallel, with a tunable timeout (that defaults to 15 seconds) if a suite takes to long to complete. This balances the desire to have sorted output with the desire to see what's happening while you are watching. On-the-fly sorting is disabled by default, and can be enabled with -PS argument to Runner .

argument to . Added the ability to specify a timeout for sorting during parallel runs. The -T parameter takes a number of seconds, which is used for test sorting. The value specified with -T plus one second is used for suite sorting.

plus one second is used for suite sorting. Added the ability to sort events locally when tests (i.e., not suites) are run in parallel via ParallelTestExecution , with a timeout if a test takes too long. Added sortingTimeout to ParallelTestExecution , the default of which is to use the timeout specified to Runner via -T (or 15 seconds if no -T given), but you can override the method. Added trait DistributedTestSorter to support this feature, and a field holding a DistributedTestSorter in Args

Enhancements for better tools integration

Added new tags CPU , Disk , Network , and reporting these as sbt's built-in resource tags, Tags.CPU , Tags.Disk , and Tags.Network .

, , , and reporting these as sbt's built-in resource tags, , , and . Added a location API for better integration with tools, especially IDEs. The location API consists of a sealed family of classes and objects, consisting of abstract base class Location , three case subclasses LineInFile , TopOfMethod , and TopOfClass , and singleton object SeeStackDepthException .

, three case subclasses , , and , and singleton object . Added a Finders annotation that specifies one or more Finders that can be used to determine selected tests from an AST in IDEs.

annotation that specifies one or more that can be used to determine selected tests from an AST in IDEs. Enhanced the ScalaTest Maven Plugin so that all of the new Runner features are accessible to Maven users.

features are accessible to Maven users. Enhanced the ScalaTest Ant Task so that all of the new Runner features are accessible to Ant users, including Gradle and Buildr users who often go through the Ant task.

features are accessible to Ant users, including Gradle and Buildr users who often go through the Ant task. Created the ScalaTest Exclipse Plugin, which is a standard part of the Scala IDE Ecosystem.

Defined the New Framework API, and integrated it both with sbt (i.e., we contributed this to sbt) and ScalaTest. The New Framework API makes all features of Runner accessible to the users of sbt, which was not possible with the original Framework API. Support for the new Framework API was released as part of sbt 0.13.

Event model enhancements

Added DiscoveryStarting and DiscoveryCompleted events, and enhanced the GUI reporter to use these events to show a spinning icon during discovery.

and events, and enhanced the GUI reporter to use these events to show a spinning icon during discovery. Added ScopeOpened and ScopeClosed events to indicate scopes, such as the scope of a describe clause in a FunSpec . Previously a “scope opened” event was indiciated via an InfoProvided events with IndentedText .

and events to indicate scopes, such as the scope of a clause in a . Previously a “scope opened” event was indiciated via an events with . Added a MarkupProvided event to indicate some markup text has been provided.

event to indicate some markup text has been provided. Added a testText field to all test events ( TestStarting , TestIgnored , TestPending , TestCanceled , TestFailed , and TestSucceeded ), primarily for integration with IntelliJ IDEA. The “test text” is either the complete test name or a suffix of it. In a FunSpec , for example, the test text will be the string passed to the it method, whereas the full test name will be the test text prefixed by the strings passed to enclosing describe method calls.

field to all test events ( , , , , , and ), primarily for integration with IntelliJ IDEA. The “test text” is either the complete test name or a suffix of it. In a , for example, the test text will be the string passed to the method, whereas the full test name will be the test text prefixed by the strings passed to enclosing method calls. Added location , an Option[Location] field, to class Event .

, an field, to class . Added a suiteId to relevant events: TestStarting , TestSucceeded , TestFailed , TestFailed , TestPending , TestCanceled , SuiteStarting , SuiteCompleted , SuiteAborted . This enables tests run previously to be dynamically tagged in future runs (because a dynamic tag requires both the testName and suiteId ).

to relevant events: , , , , , , , , . This enables tests run previously to be dynamically tagged in future runs (because a dynamic tag requires both the and ). Added a recordedEvents field to test completion events: TestPending , TestSucceeded , TestFailed , and TestCanceled . Added sealed trait RecordedEvent , which extends Event and has two subtraits: InfoProvided and MarkupProvided . Prior to 2.0, InfoProvided events were recorded during tests and played back after, so that they could appear after the test text in the standard out reporter, and in a color that matched that of the test text. The color of the test text (green if passed, red if failed, yellow if pending, etc.) could not be known until after the test had completed. By sending recorded events along with the test completion event, rather than playing them back afterwords, it is easier for Reporter s to figure out when the last event for a test has been received.

Miscellaneous enhancements

Added the ScalaTest Selenium DSL.

Added an HTML reporter. Here's an example HTML report.

To make thread dumps more clear, gave useful names to ScalaTest threads: The main thread is now ScalaTest-main . The run thread is ScalaTest-run . The dispatcher thread is ScalaTest-dispatcher . Thread-pool threads for parallel execution are ScalaTest-N . And main or thread-pool threads running test suites are renamed to ScalaTest-running-<name of suite> , then changed back again once the suite has completed.

. The run thread is . The dispatcher thread is . Thread-pool threads for parallel execution are . And main or thread-pool threads running test suites are renamed to , then changed back again once the suite has completed. Added trait NoArg to support a relatively rare use case motivated by Akka's TestKit.

to support a relatively rare use case motivated by Akka's TestKit. Added Slow tag for marking slow tests.

tag for marking slow tests. Added trait TimesOnInt , which allows you to repeatedly execute a block of code an integer number of times via syntax such as, 3 times println .

, which allows you to repeatedly execute a block of code an integer number of times via syntax such as, . Added Or and Every to ScalaUtils, allowing you to represent errors in production code as “alternate return values” (like Either) and to optionally accumulate errors.

Potential breakages

Although we have worked hard to ensure the vast majority of ScalaTest 1.x code to compile under 2.0, ScalaTest 2.0 does include a few enhancements that can break existing code. To see the list, and learn how to migrate your code if affected, see the migration guide.

New deprecations

Deprecated the before / afterEach(TestData) methods in BeforeAndAfterEach in favor of BeforeAndAfterEachTestData .

/ methods in in favor of . Deprecated FailureOf in favor of OutcomeOf .

in favor of . Deprecated expect and expectResult in Assertions in favor of assertResult .

and in in favor of . Deprecated the use of Suite as a style trait in favor of newly added Spec .

as a style trait in favor of newly added . Deprecated the assert and assume methods whose === and !== operator return Option[String] in favor of assert and assume macros that return Boolean .

and methods whose and operator return in favor of and macros that return . Deprecated the plusOrMinus operator used with Matchers in favor of the +- symbol.

operator used with in favor of the symbol. Deprecated ShouldMatchers and MustMatchers , both members of package org.scalatest.matchers , have been deprecated in favor of Matchers , which resides in package org.scalatest . ( MustMatchers was actually already deprecated in ScalaTest 1.9.2.) For folks using ShouldMatchers , getting rid of the deprecation warning should be as simple as replacing org.scalatest.matchers.ShouldMatchers with org.scalatest.Matchers . For folks using MustMatchers , however, it will unfortnately be slightly more trouble, because the new Matchers trait only supports should not must . So you will need to search and replace your uses of must with should . MustMatchers and must will continue to work for a good long deprecation period, but eventually it will be removed to make way for must possibly coming back later to serve a different purpose. Apologies for this rather large deprecation. Update: MustMatchers was resurrected in 2.1.0 in the org.scalatest package, so there's no need anymore to change must to should . In fact, the org.scalatest.MustMatchers type alias was accidentally left undeprecated in 2.0, so where you are mixing in trait org.scalatest.matchers.MustMatchers , you can in 2.0 get rid of the deprecation warning by mixing in org.scalatest.MustMatchers instead.

and , both members of package , have been deprecated in favor of , which resides in package . ( was actually already deprecated in ScalaTest 1.9.2.) For folks using , getting rid of the deprecation warning should be as simple as replacing with . For folks using , however, it will unfortnately be slightly more trouble, because the new trait only supports not . Deprecated the beforeAll and afterAll methods of trait BeforeAndAfterAll that take a config map in favor of newly added BeforeAndAfterAllConfigMap .

and methods of trait that take a config map in favor of newly added . Deprecated given / when / then / and methods of trait GivenWhenThen in favor of capitalized forms, Given / When / Then / And , because then has been deprecated as an identifier in Scala 2.10.

/ / / methods of trait in favor of capitalized forms, / / / , because has been deprecated as an identifier in Scala 2.10. Deprecated the two previously existing apply methods on Filter , and added two new ones that take an additional suiteId parameter.

methods on , and added two new ones that take an additional parameter. Deprecated trait AbstractSuite in favor of newly added trait SuiteMixin , to serve as the new base class for stackable traits that can be mixed into Suite .

in favor of newly added trait , to serve as the new base class for stackable traits that can be mixed into . Deprecated the apply method on Distributor in favor of a newly added overloaded apply method that takes a Args . During the deprecation period, the old form will call the new form with default args. The purpose of this change was to make more information available to Distributor implementations.

Expired deprecations

Removed BeforeAndAfterEachFunctions and BeforeAndAfterAllFunctions , which had been deprecated since ScalaTest 1.6.1. If you haven't done so already, you'll need to use BeforeAndAfter instead of BeforeAndAfterEachFunctions , and BeforeAndAfterAll instead of BeforeAndAfterAllFunctions .

and , which had been deprecated since ScalaTest 1.6.1. If you haven't done so already, you'll need to use instead of , and instead of . Removed MultipleFixtureFeatureSpec , MultipleFixtureFlatSpec , MultipleFixtureFreeSpec , MultipleFixtureFunSuite , MultipleFixturePropSpec , MultipleFixtureSpec , and MultipleFixtureWordSpec , which had been deprecated since ScalaTest 1.6.1. If you haven't done so already, you'll need to mix in ConfigMapFixture to a fixture.X trait instead.

, , , , , , and , which had been deprecated since ScalaTest 1.6.1. If you haven't done so already, you'll need to mix in to a trait instead. Removed org.scalatest.SuperSuite , which had been deprecated since ScalaTest 1.5. Please use Suites instead.

, which had been deprecated since ScalaTest 1.5. Please use instead. Removed the deprecated implicit conversion in the Stopper companion object that converted a Stopper to function type () => Boolean . This implicit conversion was added when the inheritance relationship between Stopper and Function0[Boolean] was severed to make it possible for Stopper to be implemented in Java. (Severing this relationship was originally a request by the IntelliJ IDEA folks, who wanted to write integration code in Java to smooth over binary incompatibilities between different Scala versions.)

companion object that converted a to function type . This implicit conversion was added when the inheritance relationship between and was severed to make it possible for to be implemented in Java. (Severing this relationship was originally a request by the IntelliJ IDEA folks, who wanted to write integration code in Java to smooth over binary incompatibilities between different Scala versions.) Removed the implicit conversion from Reporter to Event => Unit function type in the Reporter companion objecgt, which had been deprecated since ScalaTest 1.5.

to function type in the companion objecgt, which had been deprecated since ScalaTest 1.5. Removed the implicit conversion from Rerunner to Function7 , which had been deprecated since ScalaTest 1.5.

Detailed history of changes

For historians, the detailed history page preverves relevant sections of the individual release notes for major 2.0 milestones and release candidates. For a quicker to read and more understandable overview, see the Highlights section above.

Acknowledgments

ScalaTest 2.0 is brought to you by Artima, Inc., where it is tended by Bill Venners, Chua Chee Seng, and George Berger. We would like to thank all our users for their suggestions and input, everyone who has contributed source code or reviewed our work-in-progress, and our financial sponsors (who prefer to remain unamed). Without your guidance and support, ScalaTest 2.0 would not have been possible.