Testar - Selective Testing Tool for Java

Testar is a tool designed to reduce the time spent running Java unit tests. It runs on top of JUnit and automatically selects individual tests to run based on what tests previously succeeded and what changes you have made to your code since then.

Additionally, Testar allows you to find out which methods a given test covers, and vice versa.

Download Testar
Project summary page

How it works

The first time Testar is invoked, it runs all tests in JUnit test suite(s) given to it, and for each testXXX() method records code coverage information: what application methods are exercised (covered) by this test. This information, along with checksums for application/test classes and methods is saved into the Test Database (TDB).

On subsequent invocations, Testar finds out (using the saved checksums) what classes and methods you have changed. Based on this data, it selects for running only those tests that exercise the updated code. It is assumed that other tests, that passed before, will pass again, since nothing has changed in the code that they exercise. Of course, if any test didn't pass before, or if a test is new, Testar will run it unconditionally.

The above algorithm implies that all the (changeable) input for the tests is in the test code. If your tests depend on input from other sources, e.g. resource files, you should give Testar names of these files. When such a file changes, Testar will rerun all or selected tests, depending on additional options that you may specify.

How much time does it save?

Time savings may vary depending, equally importantly, on:

Generally, the smaller your changes are (or the more frequently you run Testar), and the more tests you have, the bigger are the relative savings. On the other hand, if there are some method(s) in your application that are exercised by all or most of the tests, any change to such a method will cause all these tests to rerun. That said, in practice for large applications the author has observed an average 60-70 per cent reduction in time to run tests.

How do I use it?

To use Testar, you will have to add the following options to the command line that invokes your test suite:

-javaagent:<Path to Testar.jar>=<Testar options>

For example, if normally you invoke your test suite as

java -classpath junit.jar:MyApp.jar junit.textui.TestRunner myapp.MyAppTests

you should modify the above line, at a minimum, as follows:

java -javaagent:Testar.jar=tdb=testar.tdb
-classpath junit.jar:MyApp.jar junit.textui.TestRunner myapp.MyAppTests

where the tdb option specifies the name of the Testar DB file, in which the tool will keep the information about your tests' code coverage.

Below is the list of all Testar options (to pass multiple options to the tool, specify them as option1=value1,option2=value2,...).

Option Description
tdb=TDB_name Specify non-standard test database (TDB) name. The default is testar.tdb
mainline_tdb=TDB_name Tell testar to use mainline TDB with the specified name, if the file exists (see below on why a mainline TDB may be useful).
rflist=file_name Specify the file that contains the list of resource (data) files or directories with resource files. Testar will keep track of updates to these files, and will rerun all or selected tests whenever any resource file changes. See more details on writing a resource file list below.
runalltests Run in "self-verification" mode: execute all tests, indicating unexpectedly failed ones (i.e. those failed tests that testar wouldn't run in its normal, selective testing mode).
verbose Print verbose output.

Important optimization - read this!

For best performance, it is important that Testar does not waste time and memory collecting coverage info for the code that changes very infrequently or not at all. This is typically true for JDK and other library code. Since large projects tend to use many third party libraries, it is important to avoid collecting coverage data for all such libraries.

However, what is a third library for one project may be the main code for another one. For this reason, Testar comes preconfigured to only avoid instrumenting known JDK code, that is, java.*, sun.* and com.sun.* packages. If you want to improve its performance further by excluding whatever third party libraries you wish, you should edit the Testar source code, namely the com.google.devtools.testar.TestarAgent.shouldCollectCoverageInfoForClass(String) method, and then use this updated version of Testar.

Using Testar with mainline Test Coverage Database

If many people work on the same project and update it frequently, the benefits of Testar usage may diminish due to the following issue. When you bring over to your client the latest updates done by other people, merging them with your own updates, Testar would rerun both the tests that cover your latest updates, as well as those that cover their latest updates. It simply cannot distinguish between the two kinds of updates. Since other developers collectively likely make a lot more updates every day than you do, you may find yourself wasting a lot of time running tests that you have nothing to do with.

However, this issue can be avoided if your project employs a "continuous build" or similar procedure, meaning that:

If such a procedure is in place, Testar can take advantage of that. The idea is to generate a fresh TDB in the continuous build on every iteration. Running tests under Testar and generating a TDB typically imposes a very modest performance overhead, on the order of 3-4 per cent. Subsequently, developers should be able to bring that "mainline" TDB into their workspace along with the corresponding updates to the code. When the mainline TDB is in the developer's workspace, they can run Testar with the mainline_tdb=TDB_name option (note that if the file does not exist, Testar silently ignores this option). It makes Testar load the mainline TDB together with the developer's own TDB, and then merge the information from the two databases. Each test whose covered methods are the same in the mainline and in the developer's workspace, is considered safe and is not executed. Thus, the only tests that Testar will rerun are those covering your own latest updates to the code.

Resource file list

Ideally, all the (changeable) input to the tests that may affect their execution, should be in the code of the tests themselves. However, for large systems it is often not easy to achieve that. Thus tests may depend on input from other sources, e.g. resource/data files. If that's the case, and a change to the data file does not always happen together with the change to the corresponding tests, you should pass to Testar, through the rflist=file_name option, the names of these data files, listed in a text file. When any of these data files changes, Testar will rerun all or selected tests, depending on additional options that you may specify.

The resource file list should have one data file or directory per line. For an ordinary file, all tests given to Testar will be reexecuted when this file changes. For a directory, all tests will be reexecuted if any file under that directory changes. However, the following additional options may be specified after the file or directory name to limit the effects of changing data files:

Option Description
-exclude file_name_pattern Exclude from checking any files under this directory, whose full path match the specified pattern. That is, changes to these files will have no effect on test reexecution
-only_for_tests_covering class_name_pattern file_name_pattern If any file under this directory that matches the given file name pattern changes, rerun only those tests that cover any class matching the given class name pattern

Here is an example resource file list:

foo/bar
foo/baz/data.txt
myresources -exclude *_messages_en.properties -only_for_tests_covering myapp.foo.* myqresources/foo/*

Running multiple test suites

If for some reason the tests for your application are grouped into multiple test suites, each of which should run in a separate JVM invocation, the overhead of Testar repeatedly loading and saving the same TDB, checking class files for updates etc. for each test suite, may become quite noticeable. It is clear, however, that it is enough to check classes for updates only once if you don't change any classes in between JVM invocations. It is also clear that you don't need to invoke Testar at all for test suites that don't have any tests to rerun.

To handle the above situaion properly, Testar has a special "multi-suite" execution mode. In this mode, it should be first invoked with the list of all test suites, and it determines which tests and test suites to rerun. On subsequent invocations for individual test suites, Testar would exit immediately if there are no tests to rerun in the given suite, and will rerun preselected tests in other suites.

To make Testar determine which tests and suites to rerun:

java -classpath Testar.jar:<normal test cp> com.google.devtools.testar.Testar
--tdb testar.tdb --determine_suites_and_save_in_file suitelist.txt
junit.textui.TestRunner Suite1 --- junit.textui.TestRunner Suite2 --- ...

Names of suites that have any tests to rerun will be saved in the suitelist.txt text file. The tests to rerun will be marked in the TDB.

Next, to make Testar run only preselected tests or exit immediately if there are none in this test suite, use the run_suite_if_listed_in_file=suitelist.txt option, e.g.

java -javaagent:Testar.jar=tdb=testar.tdb,run_suite_if_listed_in_file=suitelist.txt
-classpath <normal test cp> junit.textui.TestRunner Suite1

Querying TDB for tests covering a method or methods covered by a test

Code coverage database generated by Testar during test execution may be queried to obtain useful information on code coverage. To run a query, invoke Testar as follows:

java -jar Testar.jar --tdb <TDB name> <query option>

Where query option may be one of the following:

Option Description
--testsformethod method Query TDB for tests that cover the given method. The method can be specified in full or reduced form, for example void foo.bar.Baz.doSomething(int, java.lang.String), or foo.bar.Baz.doSomething(int, java.lang.String), or foo.bar.Baz.doSomething. If no method parameters are specified and method is overloaded, the data for all methods with the given class and name will be printed.
--methodsfortest test Query TDB for methods covered by the given test. The test should be specified in the format that JUnit TestCase.toString() uses: testMethodName(testClassName), e.g. --methodsfortest "testBazAction(foo.bar.BigBazTest)"

A few words on internal design

For those who are interested, here are some design details, in no specific order.

Testar uses bytecode instrumentation (AKA rewriting, manipulation) to collect code coverage information. A call to a very short static "tracker" method in Testar is injected into every method of application code. A "tracker" method registers the fact that a given application method has been called at least once. The org.objectweb.asm bytecode manipulation library is used to instrument application classes.

To get access to class file bytes before they are loaded by the VM, so that they can be rewritten, we use the mechanism described in the documentation for the java.lang.instrument package, that is available in the JDK starting from version 1.5. In essense, this mechansim allows the programmer to activate a JVM-internal class loading hook. This hook is then called after class file bytes for any class are loaded by the JVM and before these bytes are parsed and the class is installed.

One difficult issue with Testar's general approach to rerunning tests is what to do about static initializers, i.e. code that is executed once per class and per JVM invocation, when a class is loaded. Since static initalizers are not called explicitly, but instead are executed "on demand", the coverage information for them is fundamentally imprecise. Basically, if there are two tests, a and b, that use class C, then whichever test runs first, will record coverage for C 's static initializer (let's call it C_clinit). However, both a and b may depend on the results of C_clinit execution (e.g. because C_clinit initializes some data that methods of C, called by a and b, use). Furthermore, C_clinit may call other methods, trigger loading of other classes and so on, and thus in theory may affect any test executed after C is loaded.

For this reason, if Testar detects that a static initializer for C has changed, it reruns all tests after C_clinit executes. However, this is time consuming, while it looks like in practice the effects of static initializers are mostly limited. Thus it may be a good idea to implement an alternative option, which upon a change to C_clinit, would make Testar rerun, for example, only tests that cover methods of class C, or even a smaller set of methods.