Share and Enjoy

Meet me at the Green Bar

David Saff — Wed, 30 Jan 2008 16:03:00 +0000

Blogging here has been fun, but I wanted a nicer place to talk without blogger.com hanging around all the time. So, from now on, meet me at the Green Bar.

TDD in academia: a brief review

David Saff — Wed, 09 May 2007 15:11:00 +0000

The new millenium has seen growing academic interest in test-driven development. Here, I'll review the papers I've seen so far, as a starting point for further research, a reference for on-line discussions, and a source for resolving bar bets. I've squashed some interesting nuances in each study regarding the exact processes compared, and I'm happy to revise if any reader feels I've mis-quoted or over-simplified.

First, a couple studies, both using students, have found either no significant difference, or just slight improvement, in quality and productivity from using TDD.

Muller and Hagner found that TDD does not accelerate the implementation, and the resulting programs are not more reliable, but TDD developers end up with better program understanding.

Pancur, Ciglaric, Trampus, and Vidmar found that a TDD student group scored slightly, but not significantly, worse in several quality measures than a non-TDD group.

Laurie Williams' group at North Carolina State has conducted several studies of TDD using professional programmers.

George and Williams find that TDD developers took 16% more time, but passed 18% more independently-created tests. They also find that non-TDD teams almost never wrote any tests.
Maximilien and Williams performed a case study at IBM that showed a TDD project had half the defect rate of a similar non-TDD project, and, again, non-TDD developers never wrote tests.
Williams, Maximilien, and Youk performed a case study showing a 40% reduction in defects in a TDD project compared to a similar non-TDD project, with similar overall productivity.

Other groups have also found productivity and quality gains from using TDD.

Erdogmus find that quality increased linearly with the number of programmer tests, that students who wrote more tests were more productive, and that TDD students write more tests.
Bhat and Nagappan found that a TDD project had twice the quality of a non-TDD project, and actually wrote tests, although writing the tests required 15% more time.
Geras, Smith, and Miller found no productivity difference between TDD and non-TDD teams, but did find fewer unplanned test failures on the TDD team.
Janzen found that a TDD team produced code that scored higher on many code metrics than non-TDD teams, and developed more features in the same time, with a similar defect rate. Again, half of the non-TDD developers never found time to write tests.

Many of these papers are well-summarized by Janzen and Saiedian, in a survey paper that takes a positive view of TDD and predicts growing acceptance. Janzen and Saiedian also suggest test-driven learning, an application of TDD to the software engineering classroom.

To draw some tentative conclusions:

For students only, although some student groups can gain productivity and quality from using TDD, it can at worst be no better than a disciplined, non-TDD approach.
All studies on professional developers show productivity or quality gains from using TDD.
In some cases, TDD teams took longer, but produced higher-quality code. Considering that the cost of finding a bug in QA and sending it back to the developers for fixing is greater than fixing it in the first place, I consider this an unqualified win for TDD.
In many cases, although all developers were encouraged to write tests, only TDD teams did. A suite of reliable unit tests has benefits beyond initial development, for catching regressions in functionality and design and as documentation to maintainers, but these benefits were not directly measured in any of the studies above.

Exploring with JUnit Factory: 103 points in the first frame

David Saff — Tue, 08 May 2007 19:54:00 +0000

In my last post, we shook out a Theory about the number of bowls and frames in a game of bowling:


 @Theory
 public void shouldBeTenFramesWithTwoRollsInEach(Game game, Bowl first,
   Bowl second) {
  assumeNotNull(game, first, second);
  assumeThat(game.isAtBeginning(), is(true));
  assumeThat(game.getPlayers().size(), is(1));
  assumeThat(first.isStrike(), is(false));
  assumeThat(second.completesSpareAfter(first), is(false));

  for (int frame = 0; frame < 10; frame++) {
   game.bowl(first);
   game.bowl(second);
  }

  assertThat(game.isGameOver(), is(true));
 }

We used JUnitFactory to find some missing datapoints and missing assumptions, and got to the point where all JUnitFactory could find were parameters that either failed the assumptions or passed the tests, which is good.

At this point, we should think about the next functionality to test. I have almost a dozen methods that I've stubbed with fake answers, but before I completely forget about this Theory, I check the test that JUnit Factory has produced to test the "passing" path through this Theory:


    public void testShouldBeTenFramesWithTwoRollsInEach() throws Throwable {
        Game STARTING_GAME = BowlingTests.STARTING_GAME;
        BowlingTests bowlingTests = new BowlingTests();
        bowlingTests.shouldBeTenFramesWithTwoRollsInEach(STARTING_GAME, BowlingTests.THREE, new Bowl(100));
        assertNotNull("bowlingTests.assume", getPrivateField(bowlingTests, "assume"));
    }

Well, bust my buffers, where did JUnit Factory come up with the idea to bowl 100 pins with one ball? I'll have to look at its league records to see if there's been similar grade inflation. This is not the perfect test we'd like to see--no Bowl should exist with over 100 pins. I could fix this directly in the code, but we need a failing test first. Our current Theory would pass with 100 as a datapoint--what I need is a new Theory.

It's tempting to test that the Bowl constructor throws an exception whenever a pinCount over 10 is passed to it. However, testing for exceptions can be obfuscated, and it's not what I necessarily want to say--I want to say that no Bowl exists with more than 10 pins bowled:


@Theory
public void maximumPinCountIsTen(Bowl bowl) {
   assumeNotNull(bowl);
   assertThat(bowl.pinCount(), lessThanOrEqualTo(10));
}

This is easily passed by having pinCount() always return, say, 5. I am being deliberately difficult here, employing what Kent Beck calls "Fake it till you make it". The correct implementation of pinCount (return the pins passed in the constructor) is obvious, but it's worth our time to notice that our current tests don't distinguish between the obviously right and obviously wrong implementations.

The reason we can get away with a fake return from pinCount is that all we've required of the method is that it return something less than or equal to 10. Let's add another Theory about the normal behavior of pinCount:


 @Theory
 public void pinCountMatchesConstructorParameter(int pinCount) {
  assertThat(new Bowl(pinCount).pinCount(), is(pinCount));
 }

To make this pass, we can now put in the obvious definition of pinCount:


 private final int pinCount;

 public Bowl(int pinCount) {
  this.pinCount = pinCount;
 }

 public int pinCount() {
  return pinCount;
 }

Now all of our theories pass on our current data points. Let's look for other datapoints using JUnit Factory. We get this excellent test (from now on, I'll edit out all of the unimportant bits of the test, leaving the name and invocation


    public void testMaximumPinCountIsTenThrowsAssertionError() throws Throwable {
       bowlingTests.maximumPinCountIsTen(new Bowl(100));
    }

This is what we were hoping for. Our theories now catch a 100-point Bowl as an error. Before going further, I need to add this as a data point:


public static ONE_HUNDRED_BOWL = new Bowl(100);

Now maximumPinCountIsTen fails. To fix this, I'll prevent the construction of Bowls that have more than 10 pins:


 public Bowl(int pinCount) {
  if (pinCount > 10)
   throw new IllegalArgumentException("At most 10 pins in one bowl");
  this.pinCount = pinCount;
 }

Now, everything falls apart. When trying to create an instance of BowlingTests, the line


public static ONE_HUNDRED_BOWL = new Bowl(100);

causes construction to fail with an IllegalArgumentException, so no tests get run. We could remove the data point, but if we were ever to regress and forget to check arguments to the Bowl constructor, this is a test that will remind us. Popper will allow us to wrap the datapoint in a method, annotated with @DataPoint. Any @DataPoint method that throws an exception is simply ignored, so we can keep this data point around in case it's ever needed again:


 @DataPoint public Bowl oneHundredBowl() { return new Bowl(100); }

Running the tests now, we get an IllegalArgumentException on pinCountMatchesConstructorParameter:


 @Theory
 public void pinCountMatchesConstructorParameter(int pinCount) {
  assertThat(new Bowl(pinCount).pinCount(), is(pinCount));
 }

Since any integer can be passed in, some of those integers will cause IllegalArgumentExceptions. However, these integers are invalid parameters. I could explicitly check that all ints coming in are between 0 and 10, but that would duplicate the logic that the constructor itself should be doing. Instead, I'll recognize that an IllegalArgumentException is a signal that the parameter is invalid:


 @Theory
 public void pinCountMatchesConstructorParameter(int pinCount) {
  try {
   assertThat(new Bowl(pinCount).pinCount(), is(pinCount));
  } catch (IllegalArgumentException e) {
   assumeNoException(e);
  }
 }

Here, assumeNoException turns an otherwise fatal exception into an assumption failure.

Now, we run the tests again to find out that we haven't supplied any valid integers as parameters to the Bowl constructor. We can easily come up with one ourselves, but let's overuse JUnit Factory instead. We generate tests, and JUnit Factory chooses the data point 0 to test pinCountMatchesConstructorParameter. We add that data point to our TheoryContainer, and find that:

Our tests pass.
JUnit Factory can only find tests for normal returns and InvalidTheoryParameterExceptions, which is good.
All the normal-return tests use sensible parameters.

Now, I'm finally ready to move on to my next bit of functionality, but I'll let this series on bowling with Popper and JUnit Factory draw to a close. For those of you following along at home, here's the final source of our Theory class. Some things to note:

We took incredibly small steps here. Most of the times that we invoked JUnit Factory, it found things we could have found ourselves. But since this is still a new habit for me, taking it slow gives me confidence I'm not missing anything
Unlike in test-only-driven development, I expect that sometimes old theories will fail as I'm incrementally solving new ones and adding new data points. This is the theories doing their job. This initial theory was very precise about how the number of bowls in a frame corresponds to whether the first bowl is a strike, but says nothing about how to determine what a strike is. When I make the definition of "strike" more precise, I may also need to add more precision to my Bowl-counting algorithm to adjust to that. In test-only-driven development, this might require me to write a new test for the new data. In test-and-theory-driven development, the new test springs forth as a new data point or definition applied to an old theory.
Popper and JUnit Factory are still evolving, and there's a lot of room for improvement in how they work together. If you have suggestions of how to make this process easier, please mail me.
This is just one example of how we can "mash-up" data from a characterization service like JUnit Factory in order to promote quality. Other examples could include picking out lines of code that are difficult for JUnit Factory to cover, in order to suggest design improvements, or looking for patterns in exception-catching tests in order to suggest better patterns of error recovery. The sky's the limit.

Exploring with JUnit Factory: not null, I assume

David Saff — Tue, 01 May 2007 17:56:00 +0000

In my last post, we used Popper to create a Theory about scoring a bowling game. The end result boiled down to roughly this:


@RunWith(Theories.class)
public class BowlingTests extends TheoryContainer {
 public static Game STARTING_GAME = new Game();
 public static Bowl THREE = new Bowl(3);
 public static Bowl FOUR = new Bowl(4);

 @Theory
 public void shouldBeTenFramesWithTwoRollsInEach(Game game, Bowl first,
   Bowl second) {
  assumeThat(game.isAtBeginning(), is(true));
  assumeThat(game.getPlayers().size(), is(1));
  assumeThat(first.isStrike(), is(false));
  assumeThat(second.completesSpareAfter(first), is(false));

  for (int frame = 0; frame < 10; frame++) {
   game.bowl(first);
   game.bowl(second);
  }

  assertThat(game.isGameOver(), is(true));
 }
}

I believe Theories are useful for any Java developer. However, so far, I've been concentrating on test-driven development (TDD). The heartbeat of TDD with Tests and Theories is similar to that with just Tests, with one essential difference.

Use automated or manual exploration to find any data points that invalidate the current Theories. If such a data point exists, add it to the currently accepted data points. Otherwise, write a focused Test or Theory for the next bit of functionality needed. At the end of this step, a Test or Theory should fail.
Change the code so that all current Tests and Theories pass.
Refactor to the best design that passes the current tests and theories.
Repeat.

We've already done step 1--since there were no Theories, I wrote a new one, above, about ten frames with two rolls. I'll do step 2 off-screen...

There. I've written the simplest code I can think of to pass this theory--we can hard-code most of the boolean answers, especially making sure that isGameOver always returns true. Running the test in Eclipse tells me it passes for all of the parameters I've considered (all three of them). However, what about the infinite number of parameters I haven't considered? I can stare at the code and consider other options, or I can just ask JUnit Factory.

JUnit Factory is a free service accessed through a free Eclipse plug-in. Its primary purpose is to generate characterization tests for domain classes. It uses static analysis, dynamic analysis, and tuned heuristics in an attempt to characterize the current behavior of your classes, especially in unanticipated circumstances.

By turning the powerful eye of JUnit Factory on a TheoryContainer, I can see automatically if there are any inputs to my theory that pass the assumptions, but fail the assertions. I've already downloaded the plug-in, and I've made sure I meet these prerequisites:

I'm okay with Agitar storing my code on their server, and it's legal for them to do so.
I don't have any @Test annotations on my methods in my test class. JUnit Factory is very clever in refusing to generate tests for any class that looks itself like a test. I've suggested this is over-restrictive, but the limitation is in place for now. (If you use Popper like I do, you have Tests and Theories mixed up together. As a stop-gap measure, you can use the @Theory annotation on a method without parameters to indicate that it should be run as a standard JUnit test, but without alerting the little censors who live inside the JUnit Factory server.)

I focus my editor on BowlingTests push Shift-F9, and in about 30 seconds, I get my first set of characterization tests. Remember that these are tests of the methods of my TheoryContainer, not of the Game or Bowl class themselves. When scanning these tests, I'm looking only for parameters and outcomes, not the assertions themselves, which are unlikely to be interesting--all of my Theory methods, remember, return void. There's usually a few that indicate proper returns from my Theory methods, and a few indicating exceptional returns. Scanning the outline, I see:


testShouldBeTenFramesWithTwoRollsInEach()
testShouldBeTenFramesWithTwoRollsInEachThrowsNullPointerException()
testShouldBeTenFramesWithTwoRollsInEachThrowsNullPointerException1()
testShouldBeTenFramesWithTwoRollsInEachThrowsNullPointerException2()

So JUnitFactory found at least one way to make the Theory pass, and three ways to make it throw a NullPointerException. Since my tests pass with my current data points, there must be new data points I need to include to find these exceptional behaviors. Let's look at the first NullPointerException test:

    
public void testShouldBeTenFramesWithTwoRollsInEachThrowsNullPointerException() throws Throwable {
        BowlingTests bowlingTests = new BowlingTests();
        try {
            bowlingTests.shouldBeTenFramesWithTwoRollsInEach(new Game(), BowlingTests.THREE, null);
            fail("Expected NullPointerException to be thrown");
        } catch (NullPointerException ex) {
            assertNull("ex.getMessage()", ex.getMessage());
            assertThrownBy(BowlingTests.class, ex);
            assertNotNull("bowlingTests.assume", getPrivateField(bowlingTests, "assume"));
        }
    }

This is annoying--JUnitFactory is passing a null Bowl to my theory. Of course, my Theory currently claims that it accepts any value of type Bowl, and null fits that description. The other two NullPointerException tests make use of another null Bowl, and a null Game.

In order to deal with this new information, we add the two new data points to our TheoryContainer:


 public static Game STARTING_GAME = new Game();
 public static Game NULL_GAME = null;
 
 public static Bowl THREE = new Bowl(3);
 public static Bowl FOUR = new Bowl(4);
 public static Bowl NULL_BOWL = null;

Running the test, it fails with a NullPointerException, as expected. Now, the theory needs to be updated to assume that the parameters are not null. Using just Popper, we can simply use an attribute of the @Theory annotation, @Theory(nullsAccepted=false). Unfortunately, JUnitFactory does not understand this attribute, so instead, we'll have to explicitly add the assumptions:


 @Theory
 public void shouldBeTenFramesWithTwoRollsInEach(Game game, Bowl first,
   Bowl second) {
  assumeThat(game, isNotNull());
  assumeThat(first, isNotNull());
  assumeThat(second, isNotNull());

  // ...

This is a common pattern when using Popper together with JUnit Factory. In order to make it as painless as possible, you can use a shorthand from Popper 0.5:


 @Theory
 public void shouldBeTenFramesWithTwoRollsInEach(Game game, Bowl first,
   Bowl second) {
  assumeNotNull(game, first, second);

  // ...

Now, generating the tests, we see the following methods:

testShouldBeTenFramesWithTwoRollsInEach() testShouldBeTenFramesWithTwoRollsInEachThrowsInvalidTheoryParameterException()

This is what we want to see: sometimes the parameters are invalid, but these are caught by our assumptions. Anything getting past our assumptions is passing the test. Excellent.

This may feel like a lot of work for a simple skeleton. However, this work will be paid off as we move forward--this Theory actually says a lot about bowling games, and as we make Game and Bowl more sophisticated, this Theory will be waiting to catch any weirdness introduced.

And, in the future, I may publish an Eclipse plug-in that will better manage this "mash-up" of Theories and JUnit Factory, for example, by automatically inserting assumeNotNull where desirable.

Bowling with Popper style

David Saff — Fri, 27 Apr 2007 20:44:00 +0000

A few months ago, I released the first version of Popper, an extension to JUnit that allows you to supplement Tests (statements about how one particular object acts on one set of inputs) with Theories (statements about how all objects meeting certain criteria act on all inputs meeting other criteria). Popper has grown up to version 0.4, and I'd like you to try it out. Yes, you.

The big idea here is being even more precise about what you test, and how you communicate it. Tests written in a JUnit style end up saying both more and less than the developer knows--Theories help you say exactly what you know.

As an example, let's work through the a unit test Kevin Lawrence has suggested for a bowling scorer. The first requirement Kevin considers is


//  2.1.1 A game of tenpins consists of ten frames. A player delivers two balls in each of the first
//  nine frames unless a strike is scored. In the tenth frame, a player delivers three balls if a
//  strike or spare is scored. Every frame must be completed by each player bowling in regular
//  order.

Here's Kevin's test:


  @Test
  public void shouldBeTenFramesWithTwoRollsInEach() {
    for(int frame = 0; frame < 10; frame++){
      game.bowl(3);
      game.bowl(4);
    }

    assertThat(game.isGameOver(), is(true));
  }

Kevin's using Hamcrest, a matcher library that allows the convenient assertThat syntax. This is good, because we will too.

This test talks about a fixture field game, which is initialized as such:


public class GameTest {
  private Game game;

  @Before
  public void createGame() {
    game = new Game();
  }
}

This is a fairly well-written JUnit test setup, and it will correctly catch a number of bugs. However, as a communication tool to a future maintainer, it falls short:

The requirement talks about each player bowling in regular order--does the default constructor of Game produce a single-player game?
Does the property of having twenty bowls left only apply right after construction, or to Games in other states?
What's special about 3 and 4? Would any other numbers do? Just those particular numbers?

When writing a theory, the strategy is to remove specifics that are not important to the behavior under consideration, and use the object's own protocol to fill in the details that are left out. First, we can remove the invocation of the default constructor from the understanding of the test--what's important is not the constructor call, but the state the game is in at the beginning of the sequence, and the number of players. We do this by making game a parameter of the method, which is now a Theory, and making assumptions about its current state:


  @Theory
  public void shouldBeTenFramesWithTwoRollsInEach(Game game) {
    assumeThat(game.isAtBeginning(), is(true));
    assumeThat(game.getPlayers().size(), is(1));
    
    for(int frame = 0; frame < 10; frame++){
      game.bowl(3);
      game.bowl(4);
    }

    assertThat(game.isGameOver(), is(true));
  }

In the future, it may be possible to have Games that are are "at the beginning", but don't result directly from constructor calls--for example, a Game loaded from an intermediate "save-game" file. This Theory will automatically apply to those Games, as well.

Next, what's special about 3 and 4? Well, they're just two numbers that indicate that neither a strike nor spare was bowled:


  @Theory
  public void shouldBeTenFramesWithTwoRollsInEach(Game game, int firstBowl, int secondBowl) {
    assumeThat(game.isAtBeginning(), is(true));
    assumeThat(game.getPlayers().size(), is(1));
    assumeThat(firstBowl, lessThan(10));
    assumeThat(firstBowl + secondBowl, lessThan(10));

    for(int frame = 0; frame < 10; frame++){
      game.bowl(firstBowl);
      game.bowl(secondBowl);
    }

    assertThat(game.isGameOver(), is(true));
  }

(Actually, we're missing the fact here that spares in the first nine frames also lead to two bowls--the original test left that fact out, and I choose to do the same in this Theory).

The line assumeThat(firstBowl + secondBowl, lessThan(10)) bothers me. It doesn't match up with the concept of "spare" from the requirements, and it likely duplicates logic that will end up in the domain soon enough. Therefore:


  @Theory
  public void shouldBeTenFramesWithTwoRollsInEach(Game game, Bowl first, Bowl second) {
    assumeThat(game.isAtBeginning(), is(true));
    assumeThat(game.getPlayers().size(), is(1));
    assumeThat(first.isStrike(), is(false));
    assumeThat(second.completesSpareAfter(first), is(false));

    for(int frame = 0; frame < 10; frame++){
      game.bowl(first);
      game.bowl(second);
    }

    assertThat(game.isGameOver(), is(true));
  }

Now, we've removed unhelpful particulars, and added some helpful domain concepts and explicit assumptions. How does this Theory get run? There's two answers. We can use this theory for validation (Does every parameter combination we've considered in the past still work?), or exploration (Is there any new parameter combination that passes the assumptions, but fails the assertions?).

For validation, the Theory method must be declared on a subclass of TheoryContainer, which causes it to be run with a custom JUnit runner. By default, the subclass is also expected to declare as constants any valid parameter values that are currently believed to pass the theory:


@RunWith(Theories.class)
public class BowlingTheories extends TheoryContainer {
  public static Game STARTING_GAME = new Game();
  public static Bowl GUTTER_BALL = new Bowl(0);
  public static Bowl STRIKE_BALL = new Bowl(10);
  public static Bowl THREE = new Bowl(3);
  public static Bowl FOUR = new Bowl(4);

  @Theory
  public void shouldBeTenFramesWithTwoRollsInEach(Game game, Bowl first, Bowl second) { 
    // as above ...
  }
}

The custom runner will try every possible combination of parameters from the set given, but it will not try anything outside that set.

Once I'm satisfied that my code passes the Theory for all the parameters I can think of myself, I'm ready for some automated exploration, to search for parameters that I haven't thought of. However, if your code is free from legal entanglements, JUnit Factory, from Agitar, works very well for exploration. More on that in the next post. Right now, here's some fun things to try:

Download the latest version of Popper, and give it a try
Mail me with feedback, or if you'd like to participate in a case study, with a hands-on tutorial
Read our paper on the background of the Theories idea.

assertThrownException

David Saff — Thu, 26 Apr 2007 21:42:00 +0000

In my previous post, I described imposterization, a pattern for which I'm finding new uses. What can imposterization do for us? Let's look at testing for thrown exceptions.

In JUnit 3, this is the standard way of testing that a method throws an exception:


public void testIndexOutOfBounds() {
 try {
   new ArrayList().get(0);
   fail("Should have thrown exception");
 } catch (IndexOutOfBoundsException e) {
   assertEquals("Index: 0, Size: 0", e.getMessage());
 }
}

When creating JUnit 4, Kent and Erich recognized that this was a very commonly repeated pattern, and created an annotation-based way of testing for a thrown exception:


@Test(expected=IndexOutOfBoundsException.class)
public void indexOutOfBounds() {
 new ArrayList().get(0);
}

There are advantages to each approach. The explicit try/catch block looks more like the expected client code, and allows assertions about more than just the type of the thrown exception. However, as with any repetition, an experienced user starts to glaze over the details of the test, and can miss important mistakes, such as:


public void testIndexOutOfBounds() {
 try {
   new ArrayList().get(0);
   // OOPS!  Now the test will pass even with no exception!
   // fail("Should have thrown exception");
 } catch (IndexOutOfBoundsException e) {
   assertEquals("Index: 0, Size: 0", e.getMessage());
 }
}

With imposterization, we can get the succinctness of the annotation-based approach while allowing arbitrary assertions about the thrown exception:


@Test public void indexOutOfBounds() {
 IndexOutOfBoundsException e = new IndexOutOfBoundsException("Index: 0, Size: 0");
 List emptyList = new ArrayList();
 assertThrownException(is(e)).when(emptyList).get(0);
}

This now looks even less like expected client code, but it almost reads like English, and provides a one-line statement of the behavior I expect. Here, I'm using imposterization as a technique for building a mini-language, just as jMock 2 does for setting expectations. This kind of imposterization I might call syntactic imposterization: the impostor only appears in the test code, in order to refer to methods of the interface. I could then use the term semantic imposterization for uses in which the impostor is used in the same context that an original implementor would be used--for example, a mock object, or a capturing decorator from test factoring.

The expression is(e) uses Matchers from the hamcrest project, giving me a lot of power to express properties of the exception that's thrown.

When developing a mini-language using syntactic imposterization, it can be tricky to make the interface comprehensible. It would be wonderful to be able to say:


 assertThat(exceptionThrownBy(emptyList.get(0)), is(equalTo(e)));

This would use the assertThat statement made famous by Joe Walnes, which I love, but unfortunately, the expressions that I'd like to talk about often have void return values, making it impossible to compile the above statement. Therefore, the verb must always go to the end, resulting in statements that sometimes read a little more like German, or perhaps Yoda. There's three ways I've noticed so far to get around this problem without making my natural language brain hurt too much:

Dick and Jane style: in this style, I create a mutable object that remembers the methods called on it, and repeat the noun to finish the thought:
```
MethodCall mc = new MethodCall();
mc.calls(emptyList).get(0);
assertThat(mc, will(throwException(is(e))));
```
Fragment style: this is like Dick and Jane, only the noun is implicitly stored in global state or the host object, making the statements stateful. This is how JMock captures expectations:
```
call(emptyList).get(0);
assertThrownException(is(e));
```
Subordinate clause style: this style, already shown in the first code example, puts the result before the verb, allowing everything to take place in one statement, if perhaps a little backwards:
```
assertThrownException(is(equalTo(e))).when(emptyList).get(0);
```

Currently, I'm mixing the types of syntax imposterization in my tests, for asserting on thrown exceptions, as above; for identifying methods to be operated upon by my custom test framework:


FunctionPointer function = new FunctionPointer();
function.pointsAt(this).getStringReturnsA(null);
Object[] stubs = oldPopulator.stubsFor(function);

for identifying prerequisite tests:


onlyIfPassing(StubValueTableTest.class).cantAddSameValueTwice();

and for generating custom matchers based on observer method calls:


public static Matcher hasId(final String id) {
ViewReferencePropertyMatcher matcher = new ViewReferencePropertyMatcher();
matcher.mustMatch(id).getId();
return matcher;
}

However, I plan to try to coalesce a couple of these uses. I'm currently in the overuse phase* of syntax imposterization, and looking to bring my code back to its most readable state.

* Thanks to Martin Fowler for identifying this phase.

Interface imposterization

David Saff — Wed, 20 Dec 2006 14:37:00 +0000

I learn all sorts of things from jMock. From jMock 1, I was infected with the idea of a fluent interface for creating mocks, and the power of Constraint objects. Lately, I've been playing with jMock 2, and it's had me thinking even more deeply about a pattern I'm for the moment calling interface imposterization.*

To explain imposterization, I'd like to make a distinction between two kinds of subtypes of a given type (for this discussion, let's consider implementations of a Java interface). An implementor of an interface satisfies all of the documented and implied contracts for that interface. An impostor of an interface is also an implementation, as far as the object runtime is concerned, but is free to violate the interface contracts, in order to learn or assert or learn a property about the code that uses it.

If you've run across mock objects in unit testing, then you've already seen one instance of imposterization. Consider constructing a mock for a BankAccount interface, in order to test a woefully primitive transaction processor, using jMock 1 syntax:

interface BankAccount {
 void deposit(int amount);
 void withdraw(int amount);
 int getBalance();
}

@Test public void readDeposit() {
 BankAccount account = mock(BankAccount.class);
 account.expects(once()).method("deposit").with(eq(1000));
 TransactionReader reader = new TransactionReader(account);
 reader.readLine("deposit 1000");
}

Here, account is an impostor of the BankAccount interface. It's not really a proper implementor, because there are all kinds of contracts, perhaps documented, and perhaps implied, that our mock BankAccount breaks. For one thing, it probably throws an expectation exception when getBalance() is called, something that no proper implementation of BankAccount would do. But we're not trying to create a general-purpose implementation--the point of our imposterization is to learn something about the readLine method: does it interact in the right way with account?

jMock 2 also uses imposterization to set expectations on mock objects. Here's the same test in jMock 2 syntax:


@Test public void readDeposit() {
 BankAccount account = mock(BankAccount.class);
 expects(new InAnyOrder() {{
   one(account).deposit(1000);
 }});
 TransactionReader reader = new TransactionReader(account);
 reader.readLine("deposit 1000");
}

Here, the result of one(account) is a different impostor, which records a method call (.deposit(1000)) as a method call expected to be called later in the test. one(account) doesn't make any pretense of being a real BankAccount. From the point of view of the intent of object-oriented design, this is almost pure evil. But it is useful, because it reduces the "meta-noise" of the test. Rather than having to invent a new language to talk about a method invocation, we can use regular Java to just invoke the method itself.**

Impostors are not a new idea to me--my thesis work on test factoring involves creating "capturing decorators" to record invocations, and mocks to replay them--both kinds of impostors that we saw above. This is done automatically to create unit tests from arbitrary program executions, and run them after program changes, without the developer having to really know what's going on under the hood.

The new idea from jMock 2 is that plain-Java interfaces for creating impostors can be quite elegant and succinct. Knowing this, more and more problems begin to suggest imposterization to me. In my next post, I'll talk about using impostors to simplify testing exception-throwing methods, and later, we'll look at using impostors to automatically generate stubs for verifying Theories.

* The name imposterization comes from the interface Imposteriser in jMock 2. However, I won't claim that the jMock authors had exactly the same idea of what imposterization means, and I'll just have to agree to disagree about the -ize vs. -ise ending, thanks to Samuel Johnson.

** The idea of using an interface impostor to record mock object expectations has been around in EasyMock (and perhaps other frameworks) for a while. The benefits of this syntax are not without costs--I have always been uncomfortable with EasyMock's two-phase mock objects, which first capture and then replay, with a global method call in the middle to switch state. That smells to me of a missed abstraction. jMock 2 also, it turns out, uses two-phase mocks under the covers, but the interface at least encourages thinking about the recording phase differently, since developers call one(account).deposit(1000) instead of directly account.deposit(1000). But I still have the same concerns that something is being missed.

New paper: The Practice of Theories

David Saff — Thu, 07 Dec 2006 16:49:00 +0000

I've been enjoying my time working with Agitar to better understand how we think about testing, and how to use those tests to make our tools smarter. We've recently written a paper on an extension to the idea of unit tests and test-driven development. Share and enjoy, and please post your thoughts.

[Copyright notice: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.]

The Practice of Theories: Adding "For-all" Statements to "There-Exists" Tests (David Saff, Marat Boshernitsan) Submitted to IEEE Software, Special Issue on Test-Driven Development.

Abstract: Traditional unit tests in test-driven development compare a few concrete example executions against the developer’s definition of correct behavior. However, a developer knows more about how a program should behave than can be expressed through concrete examples. These general insights can be captured as theories, which precisely express software properties over potentially infinite sets of values. Combining tests with theories allows developers to say what they mean, and guarantee that their code is intuitively correct, with less effort. The consistent format of theories enables automatic tools to generate or discover values that violate these properties, discovering bugs that developers didn't think to test for.

Hamcrest and generics

David Saff — Wed, 06 Dec 2006 11:53:00 +0000

I like using hamcrest in my tests, but it has depressed me more than once how Java's generic typing rules force hamcrest to make odd typing choices. I've never investigated thoroughly, however. Luckily, someone else has.

BeforeClass and AfterClass on Suites

David Saff — Wed, 09 Aug 2006 15:13:00 +0000

The documentation of JUnit 4 does leave out some feature gems, including this:


@RunWith(Suite.class)
@SuiteClasses({ATest.class, BTest.class, CTest.class})
public class MyTests() {
   @BeforeClass public void setupDatabaseBeforeEverything() { ... }
   @AfterClass public void teardownDatabaseAfterEverything() { ... }
}

This will run


setupDatabaseBeforeEverything
ATest
BTest
CTest
teardownDatabaseAfterEverything

Meeting JUnit at a party

David Saff — Thu, 03 Aug 2006 03:50:00 +0000

[Actually, I'm still car-trip-lagged enough that I actually thought of two metaphors for why you might want to try JUnit on a green field project first. I didn't think this one nearly as well, but it's interesting enough that someone might be able to make good on what I'm trying to say...]

I think there's something deep in the summary message on the JUnit project page on SourceForge (I didn't write it, so I can praise it):

"JUnit is a simple framework for writing and running automated tests. As a political gesture, it celebrates programmers testing their own software." There is something political about JUnit. JUnit has an agenda. JUnit wants programmers to be able to test their own software. When JUnit finds itself in a room with objects that are cleanly encapsulated, each with its own responsibility, and clear ideas as to the right and wrong behavior in any given situation, JUnit can let its hair down, get to work, and have some fun.

However, there are other things to value than "programmers testing their own software". One example that's going on 10 years old is "programmers quickly producing dynamic database-backed web pages". At these parties, code can often be embedded directly into HTML, database connections are provided by implicit containers, and important functionality is put in global functions so it's available immediately wherever it's needed. JUnit goes to these parties, too, but at first, things can be a little uncomfortable.

Taking JUnit on a first date.

David Saff — Thu, 03 Aug 2006 03:43:00 +0000

A recent post to the JUnit mailing list has me thinking. I got into JUnit because I really liked writing test cases. But I really respect some people that have been really frustrated with JUnit and decided there are better uses of their time than gung-ho unit testing. Is there more going on here than personal preference? I think there might be.

Have you tried working with JUnit on a simple, non-database-backed, non-web-fronted application? Perhaps a personal toy, like Sudoku or currency conversion? Here is a place JUnit can easily shine--you have a green field to build cleanly encapsulated objects with clear responsibilities. It's on code like this that I learned to love test-driven development, and JUnit for doing TDD in Java.

If you try something like that, and you still don't like it, then move on. I do not recommend slogging away with something that makes you suffer just because someone else said it was a good idea. (Unless that person's your boss, in which case I'm sorry, and please hang around here for some concrete advice.) Come back to it in a year or so.

If, however, you do like it, then when you come back to your "real-world" tasks, with database integration, JSP web pages, fuzzy specifications, and everything else that we all know is out there, maybe you'll find ways to make these tasks a little more like the green field "toys" you started with. Or maybe you'll at least have gained some motivation to spend some more time trying to test some of the harder components.

I've heard complaints that many of the JUnit tutorial material ignores these "real world" issues, and there's some point to that. Although it's not necessary for databases and web frameworks to be test-adverse (I think that Ruby on Rails shows this pretty well), many are, and we could do a better job pointing to the excellent concrete resources out there for resolving these conflicts. On the other hand, gung-ho unit testing is a delicate new relationship. The fact that a first date happens over candlelight and Chianti doesn't mean the relationship won't eventually survive a 17-hour car trip with two cranky young children. But first things first, perhaps?

junit.sourceforge.net updated

David Saff — Sun, 12 Mar 2006 15:30:00 +0000

We've updated the homepage and some of the documentation at junit.sourceforge.net. This should make it easier to distinguish the JUnit 3.8.x documentation from the JUnit 4.x docs. Let me know if anything looks confusing. Share and Enjoy!

Getting started with JUnit 3.8.2

David Saff — Wed, 08 Mar 2006 21:51:00 +0000

There will be cool things posted here soon about JUnit 4.0, and actually 4.1, which will be released as soon as SourceForge gets its servers back in shape. Trust me. In the meantime, there's another problem I need to address.

Recently, we released JUnit 4.0, which uses annotations (from Java 5), years of added experience (from Kent and Erich), and some new ideas which could never work (from me) to provide the simplest and best unit test development framework out there. Seriously. Try it. It's very compatible: JUnit 4 tests can be run by JUnit 3 runners, and JUnit 3 tests can be run by JUnit 4 runners.

However, it turns out that

Kent and Erich had put time into a more incremental release after JUnit 3.8.1.
Not every developer and manager in the world is ready for Java 5, even if it allows them to use the simplest and best unit test development framework out there.

So, we released JUnit 3.8.2, and the world is good. However, this creates an awkward situation. We now have two versions of JUnit out there, but only one body of online documentation. So most online resources are either applicable only to JUnit 4 (The FAQ, for example), or mostly to JUnit 3 (junit.sourceforge.net, for example, although everything said there also applies to the JUnit-3-compatible part of JUnit 4).

This situation will persist until more effort can be put into reviewing and clarifying the documentation. In the meantime, I'll try to post snippets of useful information here. For example, here's how to get started with JUnit 3.8.2:

Download it
Unzip it
From the junit3.8.2 directory, run the tests by running

java -classpath .;junit.jar junit.textui.TestRunner junit.samples.AllTests

Share and enjoy!

Continuous Testing and Cruise Control

David Saff — Wed, 29 Jun 2005 08:32:00 +0000

I'm currently the primary developer of the Continuous Testing plug-in for Eclipse. I've been asked several times to explain the difference between continuous testing and the CruiseControl tool for continuous integration. Here's how I explained it recently:

What is the difference between continuous testing and cruise control?

Cruise control is a software package that implements the process of continuous integration. In the most famous article on the subject, Martin Fowler calls continuous integration"a fully automated build and test process that allows a team to build and test their software many times a day," and discusses how the CruiseControl framework was built and used at ThoughtWorks to aid in continuous integration. The idea is that once the build and test process is automated, the best way to make sure that it's run "many times a day", and to get the benefits of doing so, is to make an automated daemon responsible for running through the process on every check-in to the repository.

At another point in the same article, Fowler states "Developers typically run some subset of the unit tests with every compile as they are writing the software. This actually speeds up the development work since the tests help to find any logic errors in the code you're working on. Then, rather than debugging, you can look at the changes since you last ran the tests. Those changes should be small and thus it's a lot easier to find the bug." The idea of continuous testing is that the best way to make sure that the unit tests are run "with every compile", and to get the benefits of doing so, is to make an automated daemon responsible for doing so.

In short, continuous integration and continuous testing are both good ideas that require either developer discipline or automated support.
CruiseControl automates continuous integration to catch errors quickly:
- for an entire team
- on a shared server
- several times a day

Our plug-in automates continuous testing to catch errors quickly:
- for a single developer
- on the developer's workstation
- several times an hour, or more

Wooba Wooba?

David Saff — Wed, 16 Feb 2005 18:44:00 +0000

Just in case you find this currently-unpublicized blog, an explanation of the name. In a Sesame Street sketch of long ago, a child asks Super Grover to her make her computer work. Grover's suggestions get more and more preposterous, ending with the suggestion that he hop up and down on one foot while saying "Wooba wooba". Eventually the girl figures out that the computer needs to be plugged in.

Hence, my tech-stuff blog.

[Editor's note, 2006 March 10: I've changed the name to Share and Enjoy. No offense meant to Super Grover, but I couldn't help tinkering.]