Fin: An alternative to JUnit

by Dale Anson, June 2002

Since this article was written in June 2002, Fin has undergone some significant changes. For the most part, the following is accurate as far as intended usage, but the code in some instances has changed. Refer to the latest release for the current code.

I have some issues with JUnit. While almost everyone, including me, wholeheartedly agrees that unit tests are valuable and necessary, I have problems with the implementation. JUnit is a mature and very nicely done suite of unit testing tools, it includes a nice GUI, a command line version, and a pretty printer for reports, among other things. I have problems with it though, problems that affect how I write code and how I package my application classes. A good tool should make work easier and not force me to change my programming style.

I'll go into each of my problems in some detail, then follow with an alternative to JUnit that while admittedly isn't as mature, goes a long way toward addressing these issues while still encouraging programmers to write tests, lots of tests. And don't get me wrong, JUnit is a solid package, written by a couple of the best in the industry. It's just that every time I write a unit test with JUnit, I have an uneasy feeling that things just aren't right.

THE PROBLEMS

The first issue is that JUnit, and the Extreme Programming paradigm, assumes that all code will have tests. The philosophy is write the test first, then write the code, working back and forth on the test and the code until the test passes and the code works as intended. In real life, this isn't the case. I've been writing code for over 15 years, I've only been doing unit tests for 2. When I read things like "if code has no automated test case written for it to prove that it works, it must be assumed not to work" (from Richard Hightower's excellent 'Java Tools for Extreme Programming'), I think "what a load of crap". This is a nice ideal to strive for, but bottom line is that the proof is in the pudding. A lot of the code that I've written has never had a single unit test written for it, yet it has performed flawlessly for years. I should assume it doesn't work because it doesn't have a unit test? Give me a break!

In the real world, developers are working on programs that were written by some one else, or written before unit tests came in vogue. In many cases, as the older code is revisited, unit tests are being written. Slowly, but surely, libraries of unit tests are being developed for that code. What is lacking in JUnit is a way to tell which classes and/or methods have tests and which don't. It's well and good to see that all test pass, but what wasn't tested? Even in a modern XP environment, there is a tendency for old habits to creep back in. I know I've caught myself doing it, and I've seen others do it too. Maybe you've done it yourself -- written the code first, then the test. The idea is hot in your head, so you knock it out. Maybe it was close to quitting time, maybe you got interrupted by a meeting, maybe you forgot to write the unit test? JUnit provides no way to tell if there is code that isn't touched by a unit test.

Generally, there is a test class per class in my application, which is the accepted way of writing unit tests, and which brings up another problem. What if I do some refactoring and decide that a class should rightly be two classes? The tests for the two classes are still in the same test class. My tests will fail, of course, until I fix the test class to deal with the relocated methods. While I'm at it, I should refactor the test class into two classes, one for each of my refactored application classes, although the JUnit API doesn't actually require this. By rights, I should have done that first, but it's easier to refactor a test class after the application class has been refactored -- after all, it is the application class that is driving the refactoring, not the test class. In the real world, it could go either way, I've seen (and will even admit) to having done both, which again leads to an opportunity to have untested code floating around in a project.

While I'm admitting to not always adhering strictly to the XP way of life, let me also admit that sometimes I'm not bright enough to figure out how to test a piece of code. Here is an example similar to some of the code accompanying this article. I've used this piece of code in quite a few places over the years, but have yet to write a unit test for it. Given its track record, I'm not going to assume it doesn't work, in fact, just the opposite. Here's the example:

   private void recurseDirectory( File dir ) {
      if ( dir == null || !dir.isDirectory() )
         return;
      File[] files = dir.listFiles();
      for ( int i = 0; i < files.length; i++ ) {
         File f = files[ i ];
         if ( f.isDirectory() )
            recurseDirectory( f );
         else
            doSomething( f );
      }
   }

I've looked at this several times over the past couple of years, and have yet to figure out how to test it. There are a lot of other pieces of code that are hard to test, for example, cases where a stream or socket is required but is hard or impossible to set up in the test environment, or where the class is a small module in a larger application where some obscure object is required to properly initialize your class for testing. I wrote a plugin for jEdit a while back, I don't know how many times I restarted jEdit to be able to get a handle to a 'View' object before I read through enough code to figure out how to set one up in my test code.

The second issue is how JUnit unit tests are built with respect to my application classes. My biggest problems are where to put the tests and testing protected and private methods. The JUnit faq answers these questions as follows:

"How do I test protected methods?
Place your tests in the same package as the classes under test. Refer to "Where should I put my test files?" for examples of how to organize tests for protected method access.

How do I test private methods?
Testing private methods may be an indication that those methods should be moved into another class to promote reusability.
But if you must...
You can use reflection to subvert the access control mechanism. If you are using JDK 1.3 or higher you can use the PrivilegedAccessor class. Examples of how to use this class are available in PrivilegedAccessorTest."

Putting my test classes in the same package as the classes to be tested doesn't sound too bad for protected methods. I do work on a commercial application or two that does not distribute the tests with the finished product (although there is something to be said for doing so), so I would need to sort out which classes get shipped and which don't. With a standard naming scheme, this is a trivial task for Ant. Putting them in a different package makes it harder to test the protected methods, but does make it easier to whack the test classes from the final distribution. And for open source projects, it is becoming standard to ship the unit tests with the source code, so this is less of an issue.

But what is that about moving private methods to another class? What was all that stuff I learned about encapsulation? There are good reasons for certain methods to be private or protected. Maybe it's beyond me, but I don't know why testing a private method means that the method should be in another class. What I do know is that very often methods that rightly should be protected or private for a properly encapsulated class are made public to support the JUnit test framework.

The bottom line is that I don't like the way JUnit makes me set up tests. They are in separate classes, and possibly separate packages, which makes it hard to test protected and private methods. The framework invites me to abandon encapsulation, and this is more wide spread than you might think. Sometimes protected methods are essential. I can think of cases where several classes are tightly coupled for good reason, making certain methods 'public' opens the possibility of the API being misused and the application to fail. This is an even bigger concern with private methods. I think I'm catching the scent of a 'design smell'.

THE SOLUTION?

So what would be better a better approach? Thinking back a number of years to my early training in C, I remember my instructor saying that it is generally best to define a variable close to where you use it. Would this make sense for unit testing as well? Would it make sense to insert my test code directly into the class it is testing? A quick check of the advantages:

And the disadvantages:

This sounds pretty good. It takes care of the problems I have with JUnit and still promotes unit testing. It turns out that the two disadvantages are fairly easy to eliminate, I'll talk more about this below. What I'm missing now is a nice framework like JUnit to implement the tests.

Thinking about how such a framework would be implemented lead me to think that an introspection scheme like JUnit would do the trick. All I need is a naming convention for test methods, like having all test method names start with "test". This leads to a disadvantage over JUnit -- I'm imposing on my application classes in a way that JUnit doesn't. JUnit makes no requirements about the classes to be tested. It only has requirements of the test classes themselves. But wait! In my plan, the test classes are the application classes, which are the classes to be tested, so maybe this isn't really a disadvantage.

ASSERTIONS

Before starting work on a new framework, I wanted to explore the possibilities offered by the new 'assert' keyword introduced in Java 1.4. This keyword allows Java to natively provide a limited form of design-by-contract style programming. Assertions are a key part of JUnit, and now Java 1.4 has added this key feature to the language itself.

Using assertions requires both compiler support, so as not to break older programs that may have used 'assert' as a variable or method name, and virtual machine support.

To tell the javac compiler to allow the assert keyword to be used, use the -source 1.4 command line option. This tells the compiler to allow Java 1.4 keywords that were not supported by earlier versions. If you're using Ant, use the source=1.4 setting in the javac task. Unfortunately, the current release version of Jikes (version 1.15) doesn't yet support the assert keyword and will generate a compile time error. Assertion will be supported in Jikes 1.16.

To tell the JVM to use assertion code, use the -enableassertions or -ea command line option. If using Ant, add a <jvmargs value="-enableassertions"/> or <jvmargs value="-ea"/> tag to the java task. If you don't add this option, the JVM will simply ignore the assertions in your code.

To check that you are actually using assertions in a class, insert this code as recommended by the API documentation:

   static {
      boolean assertsEnabled = false;
      assert assertsEnabled = true; // Intentional side effect!!!
      if ( !assertsEnabled )
         throw new RuntimeException( "Asserts must be enabled!!!" );
   }

In fact, having this code in your class will REQUIRE that assertion be turned on, so this should probably be a temporary thing, not a permanent fixture.

Here's an example of using an assertion in some test code. First, here's a method to test. This method is part of a FileFilter implementation and gets the extension, minus the dot, of a file.

   public String getExtension( File f ) {
      if ( f != null ) {
         String filename = f.getName();
         int i = filename.lastIndexOf( '.' );
         if ( i > 0 && i < filename.length() - 1 ) {
            return filename.substring( i + 1 );
         }
      }
      return null;
   }
And here's a method that tests the above method:

   
   private void testGetExtension() {
      assert getExtension( new File( System.getProperty("user.home"), "index.html" ) ).equals( "html" );
   }
If the assertion fails, an AssertionError will be thrown. Of course, this isn't a complete test, the boundary conditions (no dot, dot is last character) and so on need to be tested as well.

After working with Java 1.4 assertions, it looks like my new framework won't need an Assert class like JUnit. The new keyword does the job just fine.

THE FRAMEWORK

Back to the JUnit framework -- I looked at having my classes implement Test, but that doesn't really do me much good. The good stuff is in TestCase, which is a class, not an interface. Sometimes I miss multiple inheritance! Looking at this problem a bit further, I realize that all I really need is a single method in each of my classes, something like

public static int runTests();

that will run all the tests in my class. To help generate some statistics, this method will return the number of tests actually run. Of course, if any one of the tests fails, then subsequent tests won't run. Continuing with the example started above, here is an example of a runTests method:

   public static int runTests() {
      ExtFileFilter eff = new ExtFileFilter( new String[] {"*.html", ".html", "html"}, "HTML Files" );
      eff.testAccept();
      eff.testGetExtension();
      eff.testAddExtension();
      eff.testGetDescription();
      return 4;
   }

This method is in the same class as those above. I can call this single method from anywhere, regardless of the accessibility of the methods being tested in the class. In this example, the getExtension method is public, but it could just as well be private and still be available for testing. It would be nice to have each of my classes implement an interface that has this method, but for complete accessibility, the method must be static, which can't be enforced by an interface definition. So my plan is to include a runTests method in every class (which is somewhat less work than writing a TestCase class), and write a program that will call this method for me.

"Fin" is this program. Why "Fin"? I don't know, it just sounds cool. Fin is a simple application, just a single class. Later, it'll probably grow to more as it would be nice to have a GUI and better reporting/logging.

Fin is a ClassLoader and doesn't rely on its parent ClassLoader for support. This makes it possible to do things like test part of an IDE from within the IDE itself, even with Fin running in the same VM.

Before I present the code, I also have to admit that I don't follow Sun's coding standard to the letter either. The differences are minor, but in my opinion, these minor differences provide better clarity concerning scope.

Classes follow the standard: MyClass
So do methods: myMethod
But not private fields: _my_field
And not local variables: my_variable

Fin has a constructor that accepts a String representing a directory:

   public Fin( String basedir ) {
      if ( basedir == null || basedir.length() == 0 )
         throw new IllegalArgumentException( "Parameter cannot be null or empty." );
      _basedir = new File( basedir );
      if ( !_basedir.exists() )
         throw new IllegalArgumentException( basedir + " doesn't exist." );

      System.out.println( "Checking classes in directory: " + _basedir.getAbsolutePath() );
      try {
         if ( _basedir.isDirectory() ) {
            recurseDirectory( _basedir ); // run the tests
         }
         showResults();
      }
      catch ( Exception e ) {
      }
   }

The given directory will be recursed, and an attempt will be made to test each class file found. Fin also has a private, no argument constructor that is used when Fin is testing itself.

You've seen this code already:

   private void recurseDirectory( File dir ) {
      if ( dir == null || !dir.isDirectory() )
         return;
      File[] files = dir.listFiles();
      for ( int i = 0; i < files.length; i++ ) {
         File f = files[ i ];
         if ( f.isDirectory() )
            recurseDirectory( f );
         else
            runTests( f );
      }
   }

This searches through the directory and attempts to run tests on each file found. The test is ran by runTests:

   private void runTests( File classfile ) {
      ++_classes_checked;
      try {
         Class c = findClass( classfile.getAbsolutePath() );
         String classname = c.getName();
         System.out.println( "Testing class: " + classname );
         Method m = getRunTestsMethod( c );
         if ( m == null ) {
            _not_tested.add( classname );
            return ;
         }
         try {
            _tested.add( classname );
            int cnt = invokeMethod( c, m );
            _tests_run += cnt;
            System.out.println( cnt + " test" + ( cnt != 1 ? "s" : "" ) +
                  " for " + classname + " complete." );
         }
         catch ( InvocationTargetException ite ) {
            String msg = parseException( ite, classname );
            if ( msg != null ) {
               _tests_failed.add( msg );
               System.out.println( msg );
            }
         }
      }
      catch ( Exception ignored ) {
      }
   }

This method attempts to load each file it finds as a Class, which, of course, will fail fairly often as there are generally image icons, properties files and the like scattered throughout the directory. Once a class is loaded, it checks for a 'public static int runTests()' method in the class. If the class doesn't have one, the class name is added to the list of classes that weren't tested. If it does, it invokes the method and accumulates the number of tests performed. A feature enhancement would to track statistics on the number of methods in a class versus the number of tests ran. A difference in these two numbers would suggest there is code that isn't being tested.

The class loader methods are text book, I don't go into them here. The invokeMethod method is also very simple, so I'll move on to the getRunTestsMethod method and its test:

   private Method getRunTestsMethod( Class c ) {
      Method[] methods = c.getDeclaredMethods();
      for ( int i = 0; i < methods.length; i++ ) {
         Method m = methods[ i ];
         String methodname = m.getName();
         if ( methodname.equals( "runTests" ) ) {
            int modifiers = m.getModifiers();
            if ( Modifier.isPublic( modifiers ) &&
                    Modifier.isStatic( modifiers ) &&
                    m.getReturnType().equals( Integer.TYPE ) &&
                    m.getParameterTypes().length == 0 ) {
               return m;
            }
         }
      }
      return null;
   }

This method gets a list of methods in the given class and checks each one until it finds one that is 'public', 'static', returns an int, is named "runTests", and takes no parameters. If it doesn't find such a method, it returns null. Next is an example of a Fin self-test:

   // #if ($keep_tests) //
   private void testGetRunTestsMethod() {
      Method m = getRunTestsMethod( getClass() );
      assert m.toString().equals( "public static int ise.fin.Fin.runTests()" );
   }
   // #end //

This test checks not only that the getRunTestsMethod works, but also that Fin meets the test standard of having a 'runTests' method. In fact, Fin has two such methods, but one of them (discussed above) takes a parameter and is private. I'll talk about those odd comments in a moment.

Fin tests itself. Here is the 'runTests' method. Note that the return value of 4 is hard coded. Without imposing a more rigid API, hard coding the test count into this method is the easiest way to return the number of tests. This does impose a small burden on the programmer, but certainly less burden than writing the tests in the first place.

   public static int runTests() {
      // setup
      Fin fin = new Fin();

      // test
      fin.testGetRunTestsMethod();
      fin.testInvokeMethod();
      fin.testParseException();
      fin.testLoadClassData();

      return 4;
   }

Nope, no test in sight for the recurseDirectory method!

To actually run Fin, there are two options, the command line and a bundled Ant task.

I mentioned above that there are two drawbacks to this Fin way of testing, but then said that they can be overcome. In fact, Fin provides two ways to prevent the classes from being larger than need be and to strip the tests from the code. The first way is where those funny comments come into play:

   
   // #if ($keep_tests) //
   private void testGetRunTestsMethod() {
      // test code snipped
   }
   // #end //

This is Velocity in action. The #if and #end are Velocity directives. The single line comments insulate the Velocity directives from being interpreted by the Java compiler. Velocity is a template engine from the Apache Software Foundation, and is part of the Jakarta project along with Ant and several other fine pieces of open source software. Velocity makes it trivial to preprocess java source files. Fin is bundled with a PreProcessor class that runs Velocity on each java source file. The details of how to use Velocity and what it does is beyond the scope of this article, please refer to http://jakarta.apache.org/velocity for details. Basically, when the preprocessor is invoked, it reads each java source file, processes it, and writes out the processed file to a different directory. Technically, you could make it write to the same directory, but the preprocessor alters your source code, so you probably won't want to do that.

Assuming the $keep_tests variable is set to 'true', the above code will look like this after preprocessing:

   // //
   private void testGetRunTestsMethod() {
      // test code snipped
   }
   // //

Just the Velocity code is removed. However, if the $keep_tests variable is set to false, after preprocessing the above code will look like this:

   // //

All that remains is the comments, the test method is removed entirely.

Fin is also bundled with an Ant task to help automate the preprocessing. A typical development cycle would compile and test using the original, un-preprocessed code until the software is ready for release. A distribution build would invoke the preprocessor to strip the test code, then compile and jar the processed source code. In this way, both of the problems are overcome: the class files are not larger and the test code is not part of the distribution.

This just touchs the surface of preprocessing with Velocity and Ant. The Velocity templating language is very flexible and very powerful, and using Velocity as a preprocessor can provide an amazing amount of flexibility in your code.

While using Velocity as a preprocessor has a number of benefits, one of which is helping remove the test cases to reduce class size, Fin provides a second way to remove the test methods. This second way also uses a package from the Jakarta project: BCEL, the Byte Code Engineering Library. This method deals directly with the compiled classes, not the source code, so is a postprocessor. Fin is bundled with a PostProcessor class that uses the BCEL to strip the test methods from compiled classes, as well as an Ant task to help automate the postprocessing. If you're not interested in other preprocessing that Velocity can provide, this might be an easier way to go.

In day to day use, I find that Fin is easier and more natural to use than JUnit. Like JUnit, it promotes writing of unit tests, and it eliminates a couple of problems I have with JUnit. Admittedly, it is not as mature or feature rich as JUnit, but it fits my style of working better. Fin is not the complete answer, however. JUnit has one other problem that Fin does not fix: testing anonymous inner classes remains difficult. I could do like the JUnit faq tried to do and say that you really shouldn't use anonymous inner classes (move them to another class and make them public), but that just wouldn't be right. Sometimes anonymous inner classes are handy and often times make the code clearer. I have found, however, that one of the first places I look for refactoring is at those anonymous inner classes. Overall, Fin, coupled with Velocity and BCEL, provides a viable alternative to JUnit.

Copyright 2002, Dale Anson