Jacks: Java compatibility testing, the open source way
An automated Java compiler regression test suite from the Jikes project

Maya Stodte
Contributing editor, developerWorks
February 2001

Contents:
 What is Jacks?
 Working with Jacks
 Developing a new regression test
 Examining results
 Why Tcl was used in writing Jacks
 Improving Java compilers everywhere
 The Jacks team
 Resources
 About the author

Maya Stodte takes a look at Jacks, the regression testing suite from the people at Jikes. She goes through the installation process on UNIX and Windows, talks about how to run the tests and interpret test results, and gives a code example from the project leader, Mo DeJong, of how to add new tests to the suite. She also talks with some of the people behind the project about Jacks and how Java compiler projects are making use of the scripting language based implementation..

Ever wish that Sun would finally make good on its promise and hand the Java Compatibility Kit over? Despairing of that ever happening, the people at the open source Jikes project have developed Jacks, a Java test suite. "Regression testing," according to Mo DeJong, the principal designer of Jacks, "is nothing new. The gcc and gdb projects make use of a Tcl-based automated regression testing framework called DejaGnu. While the DejaGnu framework is very powerful, it is also rather complex and a bit inflexible. DejaGNU has a number of features that are just not needed for something simple like testing a Java compiler. In developing Jacks, I wanted to keep the good ideas in DejaGnu without carrying along the extra baggage."

What is Jacks?
The Jacks test suite checks a Java compiler's conformance to the JLS (Java Language Specification). It's made up of a large number of small test cases, with each test focused on a specific section of the JLS. Eric Blake, a principal contributor to the Jacks project, describes the benefits of this type of testing in terms of its detail-oriented scope. "By generating small test cases with specified compilation behavior, then automating the execution of each of these cases, a compiler writer or debugger can quickly pinpoint problems in the translation of Java source to bytecodes."

The concept behind the development of Jacks is to simplify running tests with multiple compilers or multiple compiler configurations (a single set of tests, for example, against the last two releases of Jikes and the JDK 1.3 release of Javac). By hand you'd have to repeatedly set environmental variables and check test results against expected results. Using Jacks, you simply cd to the directory that holds the tests, invoke the Jacks framework, and indicate which compiler configuration should be used.

Sun's broken promises to Java developers helped motivate the Jikes team to get the Jacks project up and running. Sun has repeatedly stated that it would turn Java, the JCK (Java Compatibility Kit), and related Java technology over to a standards body. Because this has yet to happen, developers working on Java projects can't use the JCK to test for regressions in day to day development. And when faced with an artificial scarcity of code because of unreasonable license restrictions, they tend to simply replace the old systems with new and better ones. This is exactly what has happened with Jacks. (Although Jacks is hosted by developerWorks, it is covered by the GPL, not the IBM Public License.)

Working with Jacks
Jacks is written in Tcl, so you want to make sure you have Tcl 8.3. (You want the 8.3 version to ensure you have the tcltest extension and Unicode support, both of which Jacks requires). You can download the installer for Windows and the RPM for Red Hat x86, or simply build from the source code. See the Resources section later in this article if you're not sure where to download from; if you're using Red Hat 7, it's likely that Tcl 8.3 is already installed.

Once Tcl is installed, you need to fetch Jacks from the CVS. Then configure Jacks by including your compiler(s) path name(s) in Jacks' _setup configuration file for the compiler(s) you want to test. You will need one _setup file for each configuration you want to support. For example, Jacks comes with a javac_setup file. You would edit that file to set the path for javac. "The hardest part," according to Eric Blake, "was figuring out how to test Jikes, since I had JIKESPATH set in my environment. But I figured out what to change in the jikes_setup configuration file and all was well."

Fetching the Jacks source code from the CVS module 
setenv CVSROOT :pserver:anoncvs@oss.software.ibm.com:/usr/cvs/jikes

cvs login 
paswsd anoncvs 

cvs checkout jacks
All code samples are taken from the Jacks home page (see Resources) unless otherwise noted.

You can use Jacks with as many compilers or compiler configurations as you like. To get rid of a configuration for a compiler, simply delete its _setup file.

Once you've fetched the source code from the CVS, you'll want to include the top Jacks directory on your path so that you can run the Jacks shell script. As a preliminary precaution, you should run the shell script with no arguments to make sure everything is configured properly. 
% jacks

If everything is working, you'll see a listing of the command line options that the Jacks script accepts. If you get an error, check that the executable tclsh8.3 can be found on your path. Windows users will need to run tclsh83 directly, and pass it the jacks.tcl argument before the normal flags. You should also consider installing the Cygwin UNIX compatibility layer so that, like UNIX users, you can run Jacks with the provided shell script. The following instructions assume you are using the shell script.

For a test example, you might want to run all the tests in a given subdirectory using the Jikes compiler with the following commands: 
% cd tests/jls/packages/package-declarations/unnamed-packages
% jacks jikes

Developing a new regression test
Developing a new Jacks test case is straightforward. "Basically," in Eric Blake's words, "you design a simple source file to test a problem, put it in the specified Jacks format, then run Jacks. If the compiler results don't match the expected result, it prints an error." Here's an example from the tutorial on adding a new test case on the Jacks home page: 
// File SynchronizedInterface.java

public synchronized interface SynchronizedInterface {}

When compiled with Jikes, the following error is generated: 
% jikes SynchronizedInterface.java 

Found 1 semantic error compiling "SynchronizedInterface.java":

     3. public synchronized interface SynchronizedInterface {}
               <---------->
*** Error: synchronized is not a valid interface modifier.

A quick look at section 9.1.1 of the JLS indicates that synchronized is not a legal modifier in this context. If you try to compile this same class with the Javac compiler from an early release of the JDK, no error is generated (this bug was fixed in later releases). 
% javac SynchronizedInterface.java

Now that the problem can be reproduced, you can add a regression test case to the Jacks test suite by:

  1. Figuring out what directory the test case should go in
  2. Writing the regression test
  3. Running the new test in the Jacks framework

The format of a regression test in the tcltest framework is: 
tcltest::test NAME DESCRIPTION {
    COMMANDS
} EXPECTED_RESULT

This is the first test for section 9.1.1 of the JLS, so the NAME is 9.1.1-1.

The test case belongs in the directory tests/jls/interfaces/interface-declarations/interface-modifiers (the location is based on the name of the JLS section).

The DESCRIPTION can be anything you want it to be.

The COMMANDS section contains any Tcl commands, but the saveas and compile methods from Jacks are all you'll need most of the time.

The saveas command takes two arguments: the file name and the data to be saved in the file: 
saveas SynchronizedInterface.java \
    {public synchronized interface SynchronizedInterface {}}

The compile command takes any number of command line arguments and passes them to the java compiler. It will return PASS, FAIL, or WARN to indicate the exit status of the compiler.

The EXPECTED_RESULT is the result you expect from the compile command.

In this interface example, the compile is expected to fail. So the complete regression test would look like this: 
tcltest::test 9.1.1-1 {should generate error on synchronized interface} {
    saveas SynchronizedInterface.java \
        {synchronized interface SynchronizedInterface {}}

    compile SynchronizedInterface.java
} FAIL

Examining results
Running the tests and checking the results is completely automated, so you can actually just sit back and watch the results roll in. The Jacks framework will recursively descend through test directories and run all of the tests it finds.

If nothing goes wrong, nothing is printed. If a test fails, a description of the failure will be printed as shown in Listing 1 by Mo Dejong. This example demonstrates a bug in Javac because the first constructor calls the second, and the second calls the first. The JLS states that this is illegal (section 8.8.5.1), so a Java compiler must signal an error if this case is detected.

Let's see how Jikes does on this same test case. In Listing 2 we'll use a feature in Jacks that allows you to pass a pattern as the third argument to the jacks script. Test cases with names that do not match the pattern will be skipped. In this trivial case, the pattern is simply the name of the test case. In this example, note how the one test case we were interested in passed and the others were skipped. The above output shows that the bug found in the Javac compiler does not exist in Jikes.

While human readable results are very useful, they can quickly become hard to manage when you have a lot of test cases to deal with. Jacks recently celebrated a major milestone because it now contains over 1,000 individual JLS test cases. With this many test cases, nobody could be expected to remember which ones passed and which ones failed at any one point in time. But fear not, Jacks includes a set of logging and test result analysis features that track test results over time. This is a critical feature, since it provides a way for Java compiler developers to track the status of bug fixes and possible regressions.

How Jacks was written, and why Tcl was used
A scripting language is the natural choice when implementing a testing suite like Jacks, and Tcl was used for a number of reasons:

  • Tcl is open source, so it'll still be around in the future.
  • It's easy to install, and scripts don't need to be compiled.
  • It's easy to read and write, and scripting languages are far easier to learn than C/C++.
  • It's easy to use the string manipulation and regular expression features.
  • It's highly portable and runs on more platforms than Java.
  • It's been used successfully in thousands of organizations over the last decade.
  • Ironically, it used to be a Sun project :)

"One of Jacks' best features", according to Mo DeJong, "is the self generating documentation. On the Jacks home page you'll find a link to a test case index page which lists all of the available test cases. It's indexed and cross-referenced in a number of useful ways. You can easily look up a test case by name or find out how well a given JLS chapter is covered by the existing tests. And Tcl's highly dynamic language features made implementing self-documenting test cases easy

To date, Jacks supports the following Java compilers:

  • JDK 1.3 (1.1 and 1.2 also work but are out of date)
  • Jikes, the Open Source Java compiler from IBM
  • Kaffe, which makes use of the Kopi compiler
  • GCJ, the Java front end to gcc

Improving Java compilers everywhere
Jacks was initially focused on providing compiler testing for the Jikes project only. The original goal was to replace a homemade test system that was created for Jikes but had been abandoned because it was too hard to set up and use. It soon became clear that if the test suite was made a bit more generic, it could also be used by the other Java compiler projects. That could lead to an increase in the number of submitted test cases. At the very least, it couldn't hurt to have other Java experts evaluating the correctness test cases.

Naturally, the Jikes project makes heavy use of Jacks, but what about the GCJ and Kopi compiler projects? Tom Tromey, a resident Java guru at Red Hat, already sees the effect of Jacks development on the GCJ project. "Jacks already has actual usefulness to the GCJ project. I run Jacks whenever I make a front end compiler change, and I regularly use Jacks to find bugs in GCJ. I've found adding tests to be trivial. The framework seems easy to use and well thought out."

After running Jacks for the first time, Thomas Graf, Project Manager for the Kopi compiler, may also have become a devotee. "The first run resulted in 169 failed tests. After applying a few fixes based on the analysis of some of the failed tests, 147 tests fail. These results are very encouraging (for Jacks:). Conclusion: the Jacks test suite is really a very valuable tool for improving the quality of our compiler!"

The Jacks team
While a number of people have contributed to the Jacks project, the majority of the code contributions have come from Mo DeJong and Eric Blake. Mo DeJong works for Red Hat Inc. While not biking around the San Francisco area, Mo works on Red Hat's Source-Navigator IDE. Eric Blake is a student at Brigham Young University. Eric works in the Configurable Computing Lab as a research programmer. Eric's favorite thing about open source software is that if you don't like it, you can fix it yourself. Both made substantial contributions to the writing of this article.

Resources

About the author
Maya Stodte is a contributing writer and editor for developerWorks. She can be reached at mstodte@pop.rcn.com.