Recently, a team I know provided another reason to avoid mixing production logging with code. They have a system that processes messages and have been asked to record all the accepted messages for later reconciliation with an upstream system. They did what most Java teams would do and logged incoming messages in the class that processes them. Then they associated a special appender with that class’s logger that writes its entries to a file somewhere for later checking. The appenders are configured in a separate XML file.

One day the inevitable happened and they renamed the message processing class during a refactoring. This broke the reference in the XML configuration and the logging stopped. It wasn’t caught for a little while because there wasn’t a test. So, lesson one is that, if it matters, there should have been a test for it. But this is a pretty rigorous team that puts a lot of effort into doing things right (I’ve seen much worse), so how did they miss it?

I think part of it is the effort required to test logging. A unit test won’t do because the structure includes configuration, and acceptance tests run slowly because loggers buffer to improve performance. And part of it is to do with using a side effect of system infrastructure to implement a service. There’s nothing in the language of the implementation code that describes the meaning of reporting received messages: “it’s just logging”.

Once again, if I want my code to do something, I should just say so…

Update: I’ve had several responses here and on other media about how teams might avoid this particular failure. All of them are valid, and I know there are techniques for doing what I’m supposed to while using a logging framework.

I was trying to make a different point—that some code techniques seem to lead me in better directions than others, and that a logging framework isn’t one of them. Once again I find that the trickiness in testing an example like this is a clue that I should be looking at my design again. If I introduce a collaboration to receive structured notifications, I can separate the concepts of handling messages and reporting progress. Once I’ve split out the code to support the reconciliation messages, I can test and administer it separately—with a clear relationship between the two functions.

None of this guarantees a perfect design, but I find I do better if I let the code do the work.

In 1968, NATO funded a conference with the then provocative title of Software Engineering. Many people feel that this is the moment when software development lost its way, but the report itself is more lively that its title suggests.

It turns out that “outside in” development, with early testing is older than we thought. Here’s a quote from the report by Alan Perlis:

I’d like to read three sentences to close this issue.

1. A software system can best be designed if the testing is interlaced with the designing instead of being used after the design.
2. A simulation which matches the requirements contains the control which organizes the design of the system.
3. Through successive repetitions of this process of interlaced testing and design the model ultimately becomes the software system itself. I think that it is the key of the approach that has been suggested, that there is no such question as testing things after the fact with simulation models, but that in effect the testing and the replacement of simulations with modules that are deeper and more detailed goes on with the simulation model controlling, as it were, the place and order in which these things are done.

It’s all out there in our history, we just have to be able to find it.

If you register with the code free1250, you’ll get a discount of 1250 DKK and Goto will donate the same amount to Computers for Charities

Some of us are then rushing to Goto Amsterdam, where I’ll be giving the talk again on Friday. Again the code free1250 will do something wonderful, but I’m not quite sure what.

I think this fits into Dave Snowden’s Cynefin framework, where he distinguishes between the ordered and unordered domains. In the ordered domain, causes lead to effects. This might be difficult to see and require an expert to interpret, but essentially we expect to see the same results when we repeat an action. In the complex, unordered domain, there is no such promise. For example, we know that flocking birds are driven by three simple rules but we can’t predict exactly where a flock will go next. Groups of people are even more complex, as conscious individuals can change the structure of a system whilst being part of it. We need different techniques for working with ordered and unordered systems, as anyone who’s tried to impose order on a gang of unruly programmers will know.

Loosely, we use rules and expert knowledge for ordered systems, the appropriate actions can be decided from outside the system. Much of the software we’re commissioned to build is about lowering the cost of expertise by encoding human decision-making. This works for, say ticket processing, but is problematic for complex domains where the result of an action is literally unknowable. There, the best we can do to influence a system is to try probing it and be prepared to respond quickly to whatever happens. Joseph Pelrine uses the example of a house party—a good host knows when to introduce people, when to top up the drinks, and when to rescue someone from that awful bore from IT. A party where everyone is instructed to re-enact all the moves from last time is unlikely to be equally successful¹. Online start-ups are another example of operating in a complex environment: the Internet. Nobody really knows what all those people will do, so the best option is to act, to ship something, and then respond as the behaviour becomes clearer.

Snowden distinguishes between “fail-safe” and “safe-fail” initiatives. We use use fail-safe techniques for ordered systems because we know what’s supposed to happen and it’s more effective to get things right—we want a build system that just works. We use safe-fail techniques for unordered systems because the best we can do is to try different actions, none of which is large enough to damage the system, until we find something that takes us in the right direction—with a room full of excitable children we might try playing a video to see if it calms them down.

At the technical level, Test-Driven Development is largely fail-safe. It allows us, amongst other benefits, to develop code that just works (for multiple meanings of “work”). We take a little extra time around the writing of the code, which more than pays back within the larger development cycle. At higher levels, TDD can support safe-fail development because it lowers the cost of changing our mind later. This allows us to take an interim decision now about which small feature to implement next or which design to choose. We can afford to revisit it later when we’ve seen the result without crashing the whole project.

Continuous deployment environments such as at Forward², on the other hand, emphasize “safe-fail”. The system is partitioned up so that no individual change can damage it, and the feedback loop is tight enough that the team can detect and respond to changes very quickly. That said, even the niftiest lean start-up will have fail-safe elements too, a sustained network failure or a data breach could be the end of the company. Start-ups that fail to understand this end up teetering on the edge of disaster.

We’ve learned a lot over the last ten years about how to tune our development practices. Test-Driven Development is no more “over” than Object-Orientation is, it’s just that we understand better how to apply it. I think our early understanding was coloured by the fact that the original eXtreme Programming project, C3, was payroll, an ordered system; I don’t want my pay cheque worked out by trying some numbers and seeing who complains³. We learned to Embrace Change, that it’s a sign of a healthy development environment rather than a problem. As we’ve expanded into less predictable domains, we’re also learning to Embrace Failure.

I’ll also be presenting an engaging rant on why we should aspire to living and working in a world where stuff just works.

If you quote the promotion code FREE100 when you sign up, QCon will give you a discount of £100 and the same amount to Crisis Charity.

He has a point of contention in that he’s doubtful about the value of our end-to-end tests. To be more precise, he’s doubtful about the value of our automated end-to-end tests, a view shared by J.B.Rainsberger, and Arlo Belshee and Jim Shore. That’s a pretty serious group. I think the answer, as always, is “it depends”.

There are real advantages to writing automated end-to-end tests. As Nat pointed out in an extended message to the mailing list for the book,

Most significantly to me, however, is the difference between “testing” end-to-end or through the GUI and “test-driving”. A lot of people who are evangelical about TDD for coding do not use end-to-end tests for driving design at the system scale. I have found that writing tests gives useful design feedback, no matter what the scale.

For example, during Arlo and Jim’s session, I was struck by how many of the “failure stories” described situations where the acceptance tests were actually doing their job: revealing problems (such as deployment difficulties) that needed to be fixed.

Automating an end-to-end test helps me think more carefully about what exactly I care about in the next feature. Automating tests for many features encourages me to work out a language to describe them, which clarifies how I describe the system and makes new features easier to test.

And then there’s scale. Pretty much anything will work for a small system (although Alan Shalloway has a story about how even a quick demonstrator project can get out of hand). For larger systems, things get complicated, people come and go, and the team isn’t quite as confident as it needs to be about where things are connected. Perhaps these are symptoms of weaknesses in the team culture, but it seems wasteful to me to take the design experience we gained while writing the features not encode it somewhere.

Of course this comes at a price. Effective end-to-end tests take skill, experience, and (most important) commitment. Not every system I’ve seen has been programmed by people who are as rigorous as Nat about making the test code expressive or allowing testing to drive the design. Worse, a large collection of badly written end-to-end tests (a pattern I’ve seen a few times) is a huge drag on development. Is that price worth paying? It (ahem) depends, and part of the skill is in finding the right places to test.

So, let me turn Brian’s final question around. What would it take to make automated end-to-end tests less scary?

This talk is about the qualities we look for in test code that keep the development “habitable.” We want to make sure the tests pull their weight by making them expressive, so that we can tell what’s important when we read them and when they fail, and by making sure they don’t become a maintenance drag themselves. We need to apply as much care and attention to the tests as we do to the production code, although the coding styles may differ. Difficulty in testing might imply that we need to change our test code, but often it’s a hint that our design ideas are wrong and that we ought to change the production code. In practice, these qualities are all related to and support each other. Test-driven development combines testing, specification, and design into one holistic activity.

I just ran it at the BBC and people seemed to like it.

If you miss this opportunity, you can always see it at QCon San Francisco.

I’m running a track at QCon in San Francisco on Friday 20th November. The topic is Technical Skills for Agile Development, and it’s about some of the technical essentials that Agile teams need to keep moving.

I’ll be presenting a session, based on material from our book, on how to live with your tests over the long term.

See you there?

This claim has been made for every technical innovation I’ve seen so far (objects, event-driven programming, etc, etc). Sometimes it’s true, but most of the time it’s about what people are used to. Michael Feathers pointed out a while ago that the Ruby community is happily exploiting techniques such as meta-programming that were traditionally regarded as needing a safe pair of hands. What’s changed is that a generation has grown up with meta-programming and doesn’t regard it as problematic. Of course, there will be a degradation in understanding as an idea rolls out beyond its originators, but there’s still some value that gets through.

Sure there’s a role for people to help the generation that is struggling to pick up a new technique, but that doesn’t mean that TDD itself will always be beyond the range of mortal developers.

public void testExecuteEven() {
  Mockery mockery = new Mockery();

  final Bar bar = mockery.mock(Bar.class);
  final Sequence sequence = new NamedSequence("sequence");

  final List<String> allParameters = new ArrayList<String>();
  final List<String> expectedParameters = new ArrayList<String>();

  for (int i = 0; i < 3; i++) {
    allParameters.add("param" + i);
    if (i % 2 == 0) {
      expectedParameters.add("param" + i);
    }
  }
  final Iterator<String> iter = expectedParameters.iterator();

  mockery.checking(new Expectations() {
   {
     while (iter.hasNext()) {
       one(bar).doIt(iter.next());
        inSequence(sequence);
      }
    }
  });

  Foo subject = new Foo();
  subject.setBar(bar);
  subject.executeEven(allParameters);
  mockery.assertIsSatisfied();
}

The intentions of the test are good, but its most striking feature is that there’s so much computation going on. This doesn’t need a new technique to make it more readable, it just needs to be simplified.

A unit test should be small and focussed enough that we don’t need any general behaviour. It just has to deal with one example, so we can make it as concrete as we like. With that in mind, we can collapse the test to this:

public void testCallsDoItOnEvenIndexedElementsInList() {
  final Mockery mockery = new Mockery();
  final Bar bar = mockery.mock(Bar.class);
  final Sequence evens = mockery.sequence("evens");

  final  List<String> params = 
    Arrays.asList("param0", "param1", "param2", "param3");

  mockery.checking(new Expectations() {{
    oneOf(bar).doIt(params.get(0)); inSequence(evens);
    oneOf(bar).doIt(params.get(2)); inSequence(evens);
  }});

  Foo subject = new Foo();
  subject.setBar(bar);
  subject.executeEven(params);
  mockery.assertIsSatisfied();
}

To me, this is more direct, a simpler statement of the example—if nothing else, there’s just less code to understand. I don’t need any loops because there aren’t enough values to justify them. The expectations are clearer because they show the indices of the elements I want from the list (an alternative would have been to put in the expected values directly). And if I pulled the common features, such as the mockery and the target object, into the test class, the test would be even shorter.

The short version of this post is: be wary of any general behaviour written into a unit test. The scope should be small enough that values can be coded directly. Be especially wary of anything with an if statement. If the data setup is more complicated, then consider using a Test Data Builder.

The end-to-end tests for our example application use the WindowLicker framework to drive the Swing user interface. Our test infrastructure starts the application up in another thread (it’s as close as we can get to running from the command line), then creates a WindowLicker driver which, eventually, creates a Java AWT Robot. It turns out (we think) that this means that we have two threads trying to load and initialise the AWT library in parallel, which hangs. Our workaround is to call a delaying method before creating the WindowLicker Driver:

private void 
makeSureAwtIsLoadedBeforeStartingTheDriverOnOSXToStopDeadlock() {
  try {
    SwingUtilities.invokeAndWait(
      new Runnable() { public void run() {} });
  } catch (Exception e) {
    throw new Defect(e);
  }
}

That’s not really what invokeAndWait() is for, but it solves our problem until we can find a better answer, and we hope that the hack is at least self-explanatory.

Does anyone have a better explanation or fix? OS/X 10.5.6, Java 1.5.0_16, White MacBook 2.4 GHz Intel Core Duo. Nat‘s Linux installation works fine.

The first striking thing was that when Willem and Marc asked who was using “Mock Objects” most everyone put their hand up (which was nice), but then only a handful also said they were thinking about Roles and Responsibilities when they did (which was frustrating). We first wrote up these ideas in our paper “Mock Roles Not Objects” and much of the difficulty we see people have with the technique of Mock Objects comes from focussing on classes rather than relationships.

As it happens, an example popped up in the rest of the session, which was run as a Coding Dojo. What was interesting to me was how the group managed to turn around its design ideas. Here’s what I can remember about how it worked out.

The domain was some kind of game, with a hero who moves around an environment slaying dragons and so forth. The first couple of stories were to do with displaying the current room, and then moving from one room to another. It was a little difficult getting started because the limitations of the event didn’t allow enough time to really drive the design from outer-level requirements, but the group managed to get started with something like:

describe Hero do
  it "should describe its surroundings" do
    hero = Hero.new(room)  
    
    room.stub!(:description).and_return("a room with twisty passages")
    
    console.should_receive(:show).with("in a room with twisty passages")
    hero.look(console)
  end
end

The expectation here says that when looking, the hero should write a text describing the room to the console. This was a place to start, but it doesn’t look right. Why is a hero attached to a room? And hero.look(console) just doesn’t read well, it’s hard to tell what it means. The tensions became clearer with the next feature, which was to have the hero move from one room to another. If we write

hero.move_to(other_room)

how can we tell that this has worked? We could ask the hero to look() again, but that means making an extra call for testing, which is not related to the intent of the test. We could ask the hero what his current room is, but that’s starting to leak into Asking rather than Telling. There may be a need for the hero to hold on to his current location, but we haven’t seen it yet.

Suddenly, it became clear that the dependencies were wrong. We already have a feature that can be told about the hero’s situation, which we can build on. If the feature were to be told about what is happening to the hero, we could use that to detect the change in room. So, our example now becomes:

describe Hero do
  it "should move to a room" do
    hero = Hero.new(console)  
    
    room.stub!(:description).and_return("a room with twisty passages")
    
    console.should_receive(:show).with("in a room with twisty passages")
    
    hero.move_to(room)
  end
end

That’s better, but it’s not finished. The term Console sounds like an implementation, not a role. Most of the sword-wielding adventurers that I know don’t know how to work a Console, but they’re quite happy to tell of their great deeds to, say, a Narrator (as David Peterson suggested). If we adjust our example we get.

describe Hero do
  it "should move to a room" do
    hero = Hero.new(narrator)  
    
    room.stub!(:description).and_return("a room with twisty passages")
    
    narrator.should_receive(:says).with("in a room with twisty passages")
    
    hero.move_to(room)
  end
end

The whole example now reads as if it’s in the same domain, in the language of a D&D game. It doesn’t refer to implementation details such as a Console—we might see that code when we get to the detailed implementation of a Narrator. Obviously, there’s a lot more we could do, for a start I’d like to see more structured messages between Hero and Narrator, but the session ran out of time at about this point.

Some lessons:

1. Naming, naming, naming. It’s the most important thing. A coherent unit of code should have a coherent vocabulary, it should read well. If not, I’m probably mixing concepts which will make the code harder to understand and more brittle to change than it needs to be.
2. When I’m about to write a test, I ask “if this were to work, who would know”. That’s the most revealing question in B/TDD. If there’s no visible effect from an event, except perhaps for changing a field in the target object, then maybe it’s worth waiting until there is a visible effect, or maybe there’s a concept missing, or maybe the structure isn’t quite right. Before writing more code, I try to make sure I understand its motivation.

Willem’s (and many other people’s) approach is slightly different. He likes to explore a bit further with the code before really sorting out the names, and he’s right that there’s a risk of Analysis-Paralysis. I do that occasionally, but my experience is that the effort of being really picky at this stage forces me to be clearer about what I’m trying to achieve, to ask those questions I really ought to have answers to, before I get in too deep.

I perform an occasional unit test after I’ve encountered a failure that I don’t want to have recur, but I rarely write the tests first. If so many experienced developers don’t write unit tests, what does that say? Maybe they would be even better developers if they followed Heinz’s advice. Maybe they don’t make many mistakes that unit tests would catch because they’re already experienced. The truth is probably somewhere in between.

What that says is that we work in a horribly inefficient industry where too many developers spend their time fixing bugs (using the debugger) sent back upstream by the testers, and it looks like there’s evidence to prove it.

via Kerry Jones

Anyone new to [TDD] should begin with a partner who is more experienced, not only because it is safer but also because one can learn from the other’s experience.

Before starting the [TDD], it is important to have a general picture of the route and how it should be [TDD]‘d. If the [TDD] is long or has a crux, you should try to pick out in advance suitable spots to rest.

Three-point contact with the [code] is an important rule in [TDD] and should be followed where possible. […] You should always have eye contact with your next [refactoring] before you start your move. Scrabbling blindly about for a [refactoring] is a waste of energy. Once started, each move should be short, decisive and smoothly executed. Long moves will tire you out quickly and beginners need to spare strength for the final section of the [TDD].

From p. 17 “Rock Climbing Basics”, Johnston and Hallden, Stackpole Books, 1995

Case studies were conducted with three development teams at Microsoft and one at IBM that have adopted TDD. The results of the case studies indicate that the pre-release defect density of the four products decreased between 40% and 90% relative to similar projects that did not use the TDD practice. Subjectively, the teams experienced a 15-35% increase in initial development time after adopting TDD.

Reading the small print, these were not “pure” TDD teams since they did a lot of the requirement and design up-front. Still, a nice data point that suggests that, taking the cost of fixes into account, it’s worth taking the time to get it right.

1) “Realizing quality improvement through test driven development: results and experiences of four industrial teams” Nachiappan Nagappan, Michael Maximilien, Thirumalesh Bhat, Laurie Williams. Empirical Software Engineering Journal, Volume 13, Number 3, pp. 289-302, Springer 2008