Recently, a team I know provided another reason to avoid mixing production logging with code. They have a system that processes messages and have been asked to record all the accepted messages for later reconciliation with an upstream system. They did what most Java teams would do and logged incoming messages in the class that processes them. Then they associated a special appender with that class’s logger that writes its entries to a file somewhere for later checking. The appenders are configured in a separate XML file.

One day the inevitable happened and they renamed the message processing class during a refactoring. This broke the reference in the XML configuration and the logging stopped. It wasn’t caught for a little while because there wasn’t a test. So, lesson one is that, if it matters, there should have been a test for it. But this is a pretty rigorous team that puts a lot of effort into doing things right (I’ve seen much worse), so how did they miss it?

I think part of it is the effort required to test logging. A unit test won’t do because the structure includes configuration, and acceptance tests run slowly because loggers buffer to improve performance. And part of it is to do with using a side effect of system infrastructure to implement a service. There’s nothing in the language of the implementation code that describes the meaning of reporting received messages: “it’s just logging”.

Once again, if I want my code to do something, I should just say so…

Update: I’ve had several responses here and on other media about how teams might avoid this particular failure. All of them are valid, and I know there are techniques for doing what I’m supposed to while using a logging framework.

I was trying to make a different point—that some code techniques seem to lead me in better directions than others, and that a logging framework isn’t one of them. Once again I find that the trickiness in testing an example like this is a clue that I should be looking at my design again. If I introduce a collaboration to receive structured notifications, I can separate the concepts of handling messages and reporting progress. Once I’ve split out the code to support the reconciliation messages, I can test and administer it separately—with a clear relationship between the two functions.

None of this guarantees a perfect design, but I find I do better if I let the code do the work.

In 1968, NATO funded a conference with the then provocative title of Software Engineering. Many people feel that this is the moment when software development lost its way, but the report itself is more lively that its title suggests.

It turns out that “outside in” development, with early testing is older than we thought. Here’s a quote from the report by Alan Perlis:

I’d like to read three sentences to close this issue.

1. A software system can best be designed if the testing is interlaced with the designing instead of being used after the design.
2. A simulation which matches the requirements contains the control which organizes the design of the system.
3. Through successive repetitions of this process of interlaced testing and design the model ultimately becomes the software system itself. I think that it is the key of the approach that has been suggested, that there is no such question as testing things after the fact with simulation models, but that in effect the testing and the replacement of simulations with modules that are deeper and more detailed goes on with the simulation model controlling, as it were, the place and order in which these things are done.

It’s all out there in our history, we just have to be able to find it.

public interface TransferFetcher {
  Transfer      fetchFor(TransferId id);
  Transfer      fetchOffsetFor(Transfer transfer);
  Set<Transfer> fetchOutstandingFor(Client client, CustomerReference reference);
  Transfer      fetchFor(CustomerReference reference);
}

This looks like a reasonable design—all the methods are to do with retrieving Transfers—but it’s odd that only one of them returns a collection of Transfers. That’s a clue.

When we looked at the class, we discovered that the fetchOutstandingFor() method has a different implementation from the other methods and pulls in several dependencies that only it needs. In addition, unlike the other methods, it has only one caller (apart from its tests, of course). It doesn’t really fit in the Fetcher implementation which is now inconsistent.

It’s easy to imagine how this method got added. The programmers needed to get a feature written, and the code already had a dependency that was concerned with Transfers. It was quicker to add a method to the existing Fetcher, even if that meant making it much more complicated, than to introduce a new collaborator. They sold a Call Option—they cashed in the immediate benefit at the cost of weakening the model. The team would be ahead so long as no-one needed to change that code.

The option got called on us. As part of our reworking, we needed to change how Transfer objects were constructed so we could handle a new kind of transaction. The structure we planned meant changing another object, say Accounts, to depend on a TransferFetcher, but the current implementation of TransferFetcher depended on Accounts to implement fetchOutstandingFor(). We had a dependency loop. We should have taken a diversion and moved the behaviour of fetchOutstandingFor() into an appropriate object, but then we had our own delivery pressures. In the end, we found a workaround that allowed us to finish the task we were in the middle of, with a note to come back and fix the Fetcher.

The cost of recovery includes not just the effort of investigating and applying a solution (which would have been less when the code was introduced) but also the drag on motivation. It’s a huge gumption trap to be making steady progress towards a goal and then be knocked off course by an unnecessary design flaw. The research described in The Progress Principal suggests that small blockers like this have a disproportionate impact compared to their size. Time to break for a cup of tea.

I believe that software quality is a cumulative property. It’s the accumulation of many small good or bad design decisions that either make a codebase productive to work with or just too expensive to maintain.

…and, right on cue, Rangwald talks about The Tyranny of the Urgent.

1) The details of the domain have been changed to protect the innocent, so please don’t worry too much about the detail.

Thanks to @aparker42 for his comments

I think there’s a description in Paul Adams’ talk about online vs. offline social networks of how Dependency Injection goes bad, particularly when using one of the many automated frameworks.

Adams describes a research subject Debbie who, in “real life” has friends and contacts from very different walks of life. She has friends from college with alternative lifestyles who post images from their favourite LA gay bars. She also trains local 10-year olds in competitive swimming. Both the college friends and swimming kids have “friended” her. She was horrified to discover that these two worlds had inadvertently become linked though her social networking account.

This is the “Facebook problem”. The assumption that all relationships are equivalent was good enough for college dorms but doesn’t really scale to the rest of the world, hence Google+. As Adams points out,

Facebook itself is not the problem here. The problem here is that these are different parts of Debbie’s life that would never have been exposed to each other offline were linked online.

Like most users, Debbie wasn’t thinking of the bigger picture when she bound the whole of her life together. She was just connecting to people she knew and commenting on some pictures of guys with cute buns.

Simile alert!

Let’s revisit the right-hand side of that illustration.

This is Nat‘s diagram for the Ports and Adapters pattern. It illustrates how some people (including us) think system components should be built, with the domain logic in the centre protected from the accidental complexity of the outside world by a layer of adapters. I do not want to have my web code inadvertently linked directly to my persistence code (or even connected to LA gay bars).

That’s the trouble with the use of DI frameworks in systems that I’ve seen, there’s only one level of relationship: get me an object from the container. When I’m adding a feature, I just want to get hold of some component—and here’s an easy way to do it. It takes a lot of rigour to step back at every access to consider whether I’m introducing a subtle link between components that really shouldn’t know about each other.

I know that most of the frameworks support managing different contexts but it seems that, frankly, that’s more thinking and organisation than most teams have time for at the beginning of a project. As for cleaning up after the fact, well it’s a good way to make a living if the company can afford it and you like solving complex puzzles. More critical, however, is that the Ports and Adapters structure is recursive. Trying to manage the environments of multiple levels of subsystem with most current containers would be, in Keith Braithwaite‘s words, “impossible and/or insane”.

new again

The answer, I believe, is to save the DI frameworks for the real boundaries of the system, the parts which might change from installation to installation. Otherwise, I gather object assembly into specialised areas of the code where I can build up the run-time structure of the system with the deft use of constructors and new. It’ll look a bit complex but no worse than the equivalent DI structure (and everyone should learn to read code that looks like lisp).

If I later find that I can’t get access to some component that I think I need, that’s not necessarily a bad thing. It’s telling me that I’m introducing a new dependency and sometimes that’s a hint that a component is in the wrong place, or that I’m trying to use it from the wrong place. The coding bump is a design feedback mechanism that I miss when I can just pull objects out of a container. If I do a good job, I should find that, most of the time, I have just the right components at the time that I need them.

— Nicolo Machiavelli, The Prince

We happen to be using the Spring database helper classes to talk to Oracle with stored procedures. It turns out that there’s a bug in the driver that means that you have to jump through a few hoops to pass values in when the input parameter type is a table. This should be equivalent to an array, but apparently it isn’t, so you have to set up the callable statement correctly. Where to do this was not obvious (to us) in the Spring framework.

Here’s an example stored procedure declaration:

CREATE TYPE VARCHARTAB IS TABLE OF VARCHAR2(255);

CREATE PACKAGE db_package {
  TYPE list_type IS TABLE OF VARCHAR2(50) INDEX BY BINARY_INTEGER;
PROCEDURE a_stored_procedure(
  table_in IN list_type
)

The table_in parameter type list_type is declared within a package, which means we can’t declare the parameter as an OracleTypes.ARRAY when setting up the statement. Instead we declare it as the type of the table contents OracleTypes.VARCHAR

class MyProcedure extends StoredProcedure {
  public MyProcedure(DataSource dataSource) {
    super(dataSource, "db_package.a_stored_procedure");
    declareParameter(new SqlParameter("table_in", 
                                      OracleTypes.VARCHAR));
    compile();
  }
  
  void call(String... values) {
    execute(withParameters(values));
  }

Here’s the money quote. When setting up the parameter, you need to provide it with a SqlTypeValue. Don’t use one of the helper base classes that come out of the box, but create an implementation directly. That gives you access to the statement, which you can cast and set appropriately.

   private Map<String, Object> withParameters(String... values) {
      return ImmutableMap.of("table_in",
                             oracleIndexTableWith(50, values));
   }

   private  <T> SqlTypeValue 
   oracleIndexTableWith(final int elemMaxLen, final T... values) {
     return new SqlTypeValue() {
       @Override
       public void setTypeValue(
         PreparedStatement statement, int paramIndex, 
         int sqlType, String typeName) throws SQLException 
      {
         ((OracleCallableStatement)statement).setPlsqlIndexTable(
            paramIndex, values, values.length, values.length,  
            sqlType, elemMaxLen);
       }
     };
   }
}

That’s it. Happy copy and paste.

public void testExecuteEven() {
  Mockery mockery = new Mockery();

  final Bar bar = mockery.mock(Bar.class);
  final Sequence sequence = new NamedSequence("sequence");

  final List<String> allParameters = new ArrayList<String>();
  final List<String> expectedParameters = new ArrayList<String>();

  for (int i = 0; i < 3; i++) {
    allParameters.add("param" + i);
    if (i % 2 == 0) {
      expectedParameters.add("param" + i);
    }
  }
  final Iterator<String> iter = expectedParameters.iterator();

  mockery.checking(new Expectations() {
   {
     while (iter.hasNext()) {
       one(bar).doIt(iter.next());
        inSequence(sequence);
      }
    }
  });

  Foo subject = new Foo();
  subject.setBar(bar);
  subject.executeEven(allParameters);
  mockery.assertIsSatisfied();
}

The intentions of the test are good, but its most striking feature is that there’s so much computation going on. This doesn’t need a new technique to make it more readable, it just needs to be simplified.

A unit test should be small and focussed enough that we don’t need any general behaviour. It just has to deal with one example, so we can make it as concrete as we like. With that in mind, we can collapse the test to this:

public void testCallsDoItOnEvenIndexedElementsInList() {
  final Mockery mockery = new Mockery();
  final Bar bar = mockery.mock(Bar.class);
  final Sequence evens = mockery.sequence("evens");

  final  List<String> params = 
    Arrays.asList("param0", "param1", "param2", "param3");

  mockery.checking(new Expectations() {{
    oneOf(bar).doIt(params.get(0)); inSequence(evens);
    oneOf(bar).doIt(params.get(2)); inSequence(evens);
  }});

  Foo subject = new Foo();
  subject.setBar(bar);
  subject.executeEven(params);
  mockery.assertIsSatisfied();
}

To me, this is more direct, a simpler statement of the example—if nothing else, there’s just less code to understand. I don’t need any loops because there aren’t enough values to justify them. The expectations are clearer because they show the indices of the elements I want from the list (an alternative would have been to put in the expected values directly). And if I pulled the common features, such as the mockery and the target object, into the test class, the test would be even shorter.

The short version of this post is: be wary of any general behaviour written into a unit test. The scope should be small enough that values can be coded directly. Be especially wary of anything with an if statement. If the data setup is more complicated, then consider using a Test Data Builder.

I wonder, however, about his style for naming roles:

interface IMakeCustomerPreferred {
  void MakePreferred();
}
interface IAddOrdersToCustomer {
  void AddOrder(Order order);
}

It took me a little while to figure it out, but to me the issue is that these interface names bind the role to the underlying implementation, or at least to a larger role. One of the things that Nat Pryce and I discuss in our book is that interfaces need refactoring too. If two interfaces are similar, perhaps there’s a common concept there and they should be collapsed—which brings us more pluggable code. This implies that roles, as described by interfaces, should aim to be context-independent. In this case, I might rename one of the interfaces to:

interface IOrderCollector {
  void AddOrder(Order order)
}

since there will be contexts in which I really don’t care that it happens to be a Customer. That said, I think Dahan has other motivations with this naming scheme, since he also uses it to control retrieval from the ORM, but there might be other ways to achieve that.

The end-to-end tests for our example application use the WindowLicker framework to drive the Swing user interface. Our test infrastructure starts the application up in another thread (it’s as close as we can get to running from the command line), then creates a WindowLicker driver which, eventually, creates a Java AWT Robot. It turns out (we think) that this means that we have two threads trying to load and initialise the AWT library in parallel, which hangs. Our workaround is to call a delaying method before creating the WindowLicker Driver:

private void 
makeSureAwtIsLoadedBeforeStartingTheDriverOnOSXToStopDeadlock() {
  try {
    SwingUtilities.invokeAndWait(
      new Runnable() { public void run() {} });
  } catch (Exception e) {
    throw new Defect(e);
  }
}

That’s not really what invokeAndWait() is for, but it solves our problem until we can find a better answer, and we hope that the hack is at least self-explanatory.

Does anyone have a better explanation or fix? OS/X 10.5.6, Java 1.5.0_16, White MacBook 2.4 GHz Intel Core Duo. Nat‘s Linux installation works fine.

The first striking thing was that when Willem and Marc asked who was using “Mock Objects” most everyone put their hand up (which was nice), but then only a handful also said they were thinking about Roles and Responsibilities when they did (which was frustrating). We first wrote up these ideas in our paper “Mock Roles Not Objects” and much of the difficulty we see people have with the technique of Mock Objects comes from focussing on classes rather than relationships.

As it happens, an example popped up in the rest of the session, which was run as a Coding Dojo. What was interesting to me was how the group managed to turn around its design ideas. Here’s what I can remember about how it worked out.

The domain was some kind of game, with a hero who moves around an environment slaying dragons and so forth. The first couple of stories were to do with displaying the current room, and then moving from one room to another. It was a little difficult getting started because the limitations of the event didn’t allow enough time to really drive the design from outer-level requirements, but the group managed to get started with something like:

describe Hero do
  it "should describe its surroundings" do
    hero = Hero.new(room)  
    
    room.stub!(:description).and_return("a room with twisty passages")
    
    console.should_receive(:show).with("in a room with twisty passages")
    hero.look(console)
  end
end

The expectation here says that when looking, the hero should write a text describing the room to the console. This was a place to start, but it doesn’t look right. Why is a hero attached to a room? And hero.look(console) just doesn’t read well, it’s hard to tell what it means. The tensions became clearer with the next feature, which was to have the hero move from one room to another. If we write

hero.move_to(other_room)

how can we tell that this has worked? We could ask the hero to look() again, but that means making an extra call for testing, which is not related to the intent of the test. We could ask the hero what his current room is, but that’s starting to leak into Asking rather than Telling. There may be a need for the hero to hold on to his current location, but we haven’t seen it yet.

Suddenly, it became clear that the dependencies were wrong. We already have a feature that can be told about the hero’s situation, which we can build on. If the feature were to be told about what is happening to the hero, we could use that to detect the change in room. So, our example now becomes:

describe Hero do
  it "should move to a room" do
    hero = Hero.new(console)  
    
    room.stub!(:description).and_return("a room with twisty passages")
    
    console.should_receive(:show).with("in a room with twisty passages")
    
    hero.move_to(room)
  end
end

That’s better, but it’s not finished. The term Console sounds like an implementation, not a role. Most of the sword-wielding adventurers that I know don’t know how to work a Console, but they’re quite happy to tell of their great deeds to, say, a Narrator (as David Peterson suggested). If we adjust our example we get.

describe Hero do
  it "should move to a room" do
    hero = Hero.new(narrator)  
    
    room.stub!(:description).and_return("a room with twisty passages")
    
    narrator.should_receive(:says).with("in a room with twisty passages")
    
    hero.move_to(room)
  end
end

The whole example now reads as if it’s in the same domain, in the language of a D&D game. It doesn’t refer to implementation details such as a Console—we might see that code when we get to the detailed implementation of a Narrator. Obviously, there’s a lot more we could do, for a start I’d like to see more structured messages between Hero and Narrator, but the session ran out of time at about this point.

Some lessons:

1. Naming, naming, naming. It’s the most important thing. A coherent unit of code should have a coherent vocabulary, it should read well. If not, I’m probably mixing concepts which will make the code harder to understand and more brittle to change than it needs to be.
2. When I’m about to write a test, I ask “if this were to work, who would know”. That’s the most revealing question in B/TDD. If there’s no visible effect from an event, except perhaps for changing a field in the target object, then maybe it’s worth waiting until there is a visible effect, or maybe there’s a concept missing, or maybe the structure isn’t quite right. Before writing more code, I try to make sure I understand its motivation.

Willem’s (and many other people’s) approach is slightly different. He likes to explore a bit further with the code before really sorting out the names, and he’s right that there’s a risk of Analysis-Paralysis. I do that occasionally, but my experience is that the effort of being really picky at this stage forces me to be clearer about what I’m trying to achieve, to ask those questions I really ought to have answers to, before I get in too deep.

RT @Jtf

The next obscure piece of evidence is Maurice Murphy’s YouTube masterclass. For those who don’t know, Maurice Murphy has been Principal Trumpet in the London Symphony Orchestra for years (nerds without an interest in orchestral music will nonetheless have heard him on the Star Wars sound tracks). What’s interesting about the video is that what they emphasise most is being musical, making sense of the melody. They have no interest in pyrotechnics and assume that the student knows the basics.

The (barely sustainable) link with master-level coding and performance, we realised, is that both are about being expressive. The master coder/performer has a point of view and they use their considerable technical skills to get it across. The code that really impresses me is almost entirely about the domain, it just reads well. Most of the code most of us see, however, is stuck a level below, I have to read through the implementation to get to the meaning.

Years ago an old friend of mine said that when she first started going to conferences she wouldn’t be impressed by some of the big names because what they said was obvious. Later, she learned that being obvious can be really hard.

public class PriceReconciler {
  private final Context context;

  public PriceReconciler(Context context) {
    this.context = context;
  }

  public boolean isAvailable(Item item, Price price) {
     Location location = 
        context.getLocationFinder().findLocationFor(item);
     SKU sku = location.lookup(item);
     return context.getCurrencyConverter()
                        .compare(price, sku.cost()) < 0;
  }
 // and so on...
}

public class PriceReconciler {
  private final LocationFinder locationFinder;
  private final CurrencyConverter currencyConverter;
  private final LooseChangeCollector looseChangeCollector;

  public PriceReconciler(Context context) {
    this.locationFinder = context.getLocationFinder();
    this.currencyConverter = context.getCurrencyConverter();
    this.looseChangeCollector = 
      context.getLooseChangeCollector();
  }
 // and so on...
}

Doing this made clear that the class used only three of the many services available on the context. More interesting, it showed that the looseChangeCollector was only used once, in a method that turned out to be referenced just once—by a class that also had access to the context. Our extraction highlighted that this behaviour was in the wrong place, it didn’t really have a relationship with the rest of PriceReconciler, so we moved it to its calling object and simplified the PriceReconciler. With just two dependencies, our next step was to set them directly in the constructor

public class PriceReconciler {
  private final LocationFinder locationFinder;
  private final CurrencyConverter currencyConverter;

  public PriceReconciler(LocationFinder locationFinder, 
                         CurrencyConverter currencyConverter) 
 {
    this.locationFinder = locationFinder;
    this.currencyConverter = currencyConverter;
  }
 // and so on...
}

which made the PriceReconclier just a little easier to reuse since we’d narrowed its dependencies.

Kent Beck wrote recently about this tension between being concrete and abstract parameters. There obviously isn’t a single answer, except to note that when two or three objects keep turning up together, there’s probably a missing intermediate concept where some of the behaviour really belongs. The trick, when turning that concept into a type, is to give it a meaningful name so that it’s hard to add features that don’t belong.