Testar - Selective Testing Tool for Java
Testar is a tool designed to reduce the time spent running Java unit tests. It runs
on top of JUnit and automatically selects individual tests to run based on what
tests previously succeeded and what changes you have made to your code since then.
Additionally, Testar allows you to find out which methods a given test covers, and vice
versa.
Download Testar
Project summary page
The first time Testar is invoked, it runs all tests in JUnit test suite(s) given to
it, and for each testXXX() method records code coverage information: what application
methods are exercised (covered) by this test. This information, along with checksums
for application/test classes and methods is saved into the Test Database (TDB).
On subsequent invocations, Testar finds out (using the saved checksums) what classes and
methods you have changed. Based on this data, it selects for running only those
tests that exercise the updated code. It is assumed that other tests, that passed
before, will pass again, since nothing has changed in the code that they exercise.
Of course, if any test didn't pass before, or if a test is new, Testar will run it
unconditionally.
The above algorithm implies that all the (changeable) input for the tests
is in the test code. If your tests depend on input from other sources, e.g. resource
files, you should give Testar names of these files. When such a file changes, Testar
will rerun all or selected tests, depending on additional options that you may specify.
Time savings may vary depending, equally importantly, on:
- The amount of changes to the code
- How "popular" is the piece of code you have changed
- How much code each test covers
- How many tests there are in your suite(s)
Generally, the smaller your changes are (or the more frequently you run Testar), and
the more tests you have, the bigger are the relative savings. On the other hand, if
there are some method(s) in your application that are exercised by all or most of
the tests, any change to such a method will cause all these tests to rerun.
That said, in practice for large applications the author has observed an average
60-70 per cent reduction in time to run tests.
To use Testar, you will have to add the following options to the command line that
invokes your test suite:
-javaagent:<Path to Testar.jar>=<Testar options>
For example, if normally you invoke your test suite as
java -classpath junit.jar:MyApp.jar junit.textui.TestRunner myapp.MyAppTests
you should modify the above line, at a minimum, as follows:
java -javaagent:Testar.jar=tdb=testar.tdb
-classpath junit.jar:MyApp.jar junit.textui.TestRunner myapp.MyAppTests
where the tdb option specifies the name of the Testar DB file, in which the
tool will keep the information about your tests' code coverage.
Below is the list of all Testar options (to pass multiple options to the tool, specify
them as option1=value1,option2=value2,...).
| Option | Description |
|---|
tdb=TDB_name | Specify non-standard test database (TDB) name. The default is testar.tdb |
mainline_tdb=TDB_name | Tell testar to use mainline TDB with the specified name, if the file exists (see below on why a mainline TDB may be useful). |
rflist=file_name | Specify the file that contains the list of resource (data) files or directories with resource files. Testar will keep track of updates to these files, and will rerun all or selected tests whenever any resource file changes. See more details on writing a resource file list below. |
runalltests | Run in "self-verification" mode: execute all tests, indicating unexpectedly failed ones (i.e. those failed tests that testar wouldn't run in its normal, selective testing mode). |
verbose | Print verbose output. |
For best performance, it is important that Testar does not waste time and memory
collecting coverage info for the code that changes very infrequently or not at all.
This is typically true for JDK and other library code. Since large projects tend to
use many third party libraries, it is important to avoid collecting coverage data for
all such libraries.
However, what is a third library for one project may be the main code for another
one. For this reason, Testar comes preconfigured to only avoid instrumenting
known JDK code, that is, java.*, sun.* and
com.sun.* packages. If you want to improve its performance further by
excluding whatever third party libraries you wish, you should edit the Testar source code,
namely the
com.google.devtools.testar.TestarAgent.shouldCollectCoverageInfoForClass(String)
method, and then use this updated version of Testar.
If many people work on the same project and update it frequently, the benefits of Testar
usage may diminish due to the following issue. When you bring over to your client the latest
updates done by other people, merging them with your own updates, Testar would rerun both the
tests that cover your latest updates, as well as those that
cover their latest updates. It simply cannot distinguish between the two kinds of
updates. Since other developers collectively likely make a lot more updates every day than
you do, you may find yourself wasting a lot of time running tests that you have nothing to
do with.
However, this issue can be avoided if your project employs a "continuous build" or similar
procedure, meaning that:
- the whole project is periodically being rebuilt and retested with all the latest updates
(usually automatically on a dedicated machine)
- you get the updates by others only after they are tested successfully
If such a procedure is in place, Testar can take advantage of that.
The idea is to generate a fresh TDB in the continuous build on every iteration. Running tests
under Testar and generating a TDB typically imposes a very modest performance overhead, on
the order of 3-4 per cent. Subsequently, developers should be able to bring that "mainline"
TDB into their workspace along with the corresponding updates to the code.
When the mainline TDB is in the developer's workspace, they can run Testar with the
mainline_tdb=TDB_name option (note that if the file does not exist, Testar
silently ignores this option). It makes Testar load the mainline TDB together with the
developer's own TDB, and then merge the information from the two databases. Each test whose
covered methods are the same in the mainline and in the developer's workspace, is considered
safe and is not executed. Thus, the only tests that Testar will rerun are those covering your
own latest updates to the code.
Ideally, all the (changeable) input to the tests that may affect their execution, should
be in the code of the tests themselves. However, for large systems it is often not easy to
achieve that. Thus tests may depend on input from other sources, e.g. resource/data files.
If that's the case, and a change to the data file does not always happen together with the
change to the corresponding tests, you should pass to Testar, through the
rflist=file_name option, the names of these data files, listed in a
text file. When any of these data files changes, Testar will rerun all or selected tests,
depending on additional options that you may specify.
The resource file list should have one data file or directory per line. For an ordinary file,
all tests given to Testar will be reexecuted when this file changes. For a directory, all
tests will be reexecuted if any file under that directory changes. However, the following
additional options may be specified after the file or directory name to limit the effects of
changing data files:
| Option | Description |
|---|
-exclude file_name_pattern | Exclude from checking any files under this directory, whose full path match the specified pattern. That is, changes to these files will have no effect on test reexecution |
-only_for_tests_covering class_name_pattern file_name_pattern | If any file under this directory that matches the given file name pattern changes, rerun only those tests that cover any class matching the given class name pattern |
Here is an example resource file list:
foo/bar
foo/baz/data.txt
myresources -exclude *_messages_en.properties -only_for_tests_covering myapp.foo.* myqresources/foo/*
If for some reason the tests for your application are grouped into multiple test suites,
each of which should run in a separate JVM invocation, the overhead of Testar repeatedly
loading and saving the same TDB, checking class files for updates etc. for each test suite,
may become quite noticeable. It is clear, however, that it is enough to check classes for
updates only once if you don't change any classes in between JVM invocations. It is also
clear that you don't need to invoke Testar at all for test suites that don't have any tests
to rerun.
To handle the above situaion properly, Testar has a special "multi-suite" execution mode.
In this mode, it should be first invoked with the list of all test suites, and it determines
which tests and test suites to rerun. On subsequent invocations for individual test suites,
Testar would exit immediately if there are no tests to rerun in the given suite, and will
rerun preselected tests in other suites.
To make Testar determine which tests and suites to rerun:
java -classpath Testar.jar:<normal test cp> com.google.devtools.testar.Testar
--tdb testar.tdb --determine_suites_and_save_in_file suitelist.txt
junit.textui.TestRunner Suite1 --- junit.textui.TestRunner Suite2 --- ...
Names of suites that have any tests to rerun will be saved in the suitelist.txt
text file. The tests to rerun will be marked in the TDB.
Next, to make Testar run only preselected tests or exit immediately if there are none
in this test suite, use the run_suite_if_listed_in_file=suitelist.txt option, e.g.
java -javaagent:Testar.jar=tdb=testar.tdb,run_suite_if_listed_in_file=suitelist.txt
-classpath <normal test cp> junit.textui.TestRunner Suite1
Querying TDB for tests covering a method or methods covered by a test
Code coverage database generated by Testar during test execution may be queried to obtain
useful information on code coverage. To run a query, invoke Testar as follows:
java -jar Testar.jar --tdb <TDB name> <query option>
Where query option may be one of the following:
| Option | Description |
|---|
--testsformethod method | Query TDB for tests that cover the given method. The method can be specified in full or reduced form, for example void foo.bar.Baz.doSomething(int, java.lang.String), or foo.bar.Baz.doSomething(int, java.lang.String), or foo.bar.Baz.doSomething. If no method parameters are specified and method is overloaded, the data for all methods with the given class and name will be printed. |
--methodsfortest test | Query TDB for methods covered by the given test. The test should be specified in the format that JUnit TestCase.toString() uses: testMethodName(testClassName), e.g. --methodsfortest "testBazAction(foo.bar.BigBazTest)" |
For those who are interested, here are some design details, in no specific order.
Testar uses bytecode instrumentation (AKA rewriting, manipulation) to collect code
coverage information. A call to a very short static "tracker" method in Testar is
injected into every method of application code. A "tracker" method registers the
fact that a given application method has been called at least once. The
org.objectweb.asm bytecode manipulation library is used to instrument application
classes.
To get access to class file bytes before they are loaded by the VM, so that they
can be rewritten, we use the mechanism described in the documentation for the
java.lang.instrument package, that is available in the JDK starting from
version 1.5. In essense, this mechansim allows the programmer to activate a JVM-internal
class loading hook. This hook is then called after class file bytes for any class
are loaded by the JVM and before these bytes are parsed and the class is installed.
One difficult issue with Testar's general approach to rerunning tests is what to do
about static initializers, i.e. code that is executed once per class and per JVM
invocation, when a class is loaded. Since static initalizers are not called explicitly, but instead
are executed "on demand", the coverage information for them is fundamentally imprecise.
Basically, if there are two tests, a and b, that use class C, then whichever test
runs first, will record coverage for C 's static initializer (let's call it C_clinit).
However, both a and b may depend on the results of C_clinit execution (e.g.
because C_clinit initializes some data that methods of C, called by a and b,
use). Furthermore, C_clinit may call other methods, trigger loading of other classes
and so on, and thus in theory may affect any test executed after C is loaded.
For this reason, if Testar detects that a static initializer for C has changed,
it reruns all tests after C_clinit executes. However, this is time consuming,
while it looks like in practice the effects of static initializers are mostly limited.
Thus it may be a good idea to implement an alternative option, which upon a change to
C_clinit, would make Testar rerun, for example, only tests that cover methods
of class C, or even a smaller set of methods.