Recently, a team I know provided another reason to avoid mixing production logging with code. They have a system that processes messages and have been asked to record all the accepted messages for later reconciliation with an upstream system. They did what most Java teams would do and logged incoming messages in the class that processes them. Then they associated a special appender with that class’s logger that writes its entries to a file somewhere for later checking. The appenders are configured in a separate XML file.

One day the inevitable happened and they renamed the message processing class during a refactoring. This broke the reference in the XML configuration and the logging stopped. It wasn’t caught for a little while because there wasn’t a test. So, lesson one is that, if it matters, there should have been a test for it. But this is a pretty rigorous team that puts a lot of effort into doing things right (I’ve seen much worse), so how did they miss it?

I think part of it is the effort required to test logging. A unit test won’t do because the structure includes configuration, and acceptance tests run slowly because loggers buffer to improve performance. And part of it is to do with using a side effect of system infrastructure to implement a service. There’s nothing in the language of the implementation code that describes the meaning of reporting received messages: “it’s just logging”.

Once again, if I want my code to do something, I should just say so…

Update: I’ve had several responses here and on other media about how teams might avoid this particular failure. All of them are valid, and I know there are techniques for doing what I’m supposed to while using a logging framework.

I was trying to make a different point—that some code techniques seem to lead me in better directions than others, and that a logging framework isn’t one of them. Once again I find that the trickiness in testing an example like this is a clue that I should be looking at my design again. If I introduce a collaboration to receive structured notifications, I can separate the concepts of handling messages and reporting progress. Once I’ve split out the code to support the reconciliation messages, I can test and administer it separately—with a clear relationship between the two functions.

None of this guarantees a perfect design, but I find I do better if I let the code do the work.

In 1968, NATO funded a conference with the then provocative title of Software Engineering. Many people feel that this is the moment when software development lost its way, but the report itself is more lively that its title suggests.

It turns out that “outside in” development, with early testing is older than we thought. Here’s a quote from the report by Alan Perlis:

I’d like to read three sentences to close this issue.

1. A software system can best be designed if the testing is interlaced with the designing instead of being used after the design.
2. A simulation which matches the requirements contains the control which organizes the design of the system.
3. Through successive repetitions of this process of interlaced testing and design the model ultimately becomes the software system itself. I think that it is the key of the approach that has been suggested, that there is no such question as testing things after the fact with simulation models, but that in effect the testing and the replacement of simulations with modules that are deeper and more detailed goes on with the simulation model controlling, as it were, the place and order in which these things are done.

It’s all out there in our history, we just have to be able to find it.

public interface TransferFetcher {
  Transfer      fetchFor(TransferId id);
  Transfer      fetchOffsetFor(Transfer transfer);
  Set<Transfer> fetchOutstandingFor(Client client, CustomerReference reference);
  Transfer      fetchFor(CustomerReference reference);
}

This looks like a reasonable design—all the methods are to do with retrieving Transfers—but it’s odd that only one of them returns a collection of Transfers. That’s a clue.

When we looked at the class, we discovered that the fetchOutstandingFor() method has a different implementation from the other methods and pulls in several dependencies that only it needs. In addition, unlike the other methods, it has only one caller (apart from its tests, of course). It doesn’t really fit in the Fetcher implementation which is now inconsistent.

It’s easy to imagine how this method got added. The programmers needed to get a feature written, and the code already had a dependency that was concerned with Transfers. It was quicker to add a method to the existing Fetcher, even if that meant making it much more complicated, than to introduce a new collaborator. They sold a Call Option—they cashed in the immediate benefit at the cost of weakening the model. The team would be ahead so long as no-one needed to change that code.

The option got called on us. As part of our reworking, we needed to change how Transfer objects were constructed so we could handle a new kind of transaction. The structure we planned meant changing another object, say Accounts, to depend on a TransferFetcher, but the current implementation of TransferFetcher depended on Accounts to implement fetchOutstandingFor(). We had a dependency loop. We should have taken a diversion and moved the behaviour of fetchOutstandingFor() into an appropriate object, but then we had our own delivery pressures. In the end, we found a workaround that allowed us to finish the task we were in the middle of, with a note to come back and fix the Fetcher.

The cost of recovery includes not just the effort of investigating and applying a solution (which would have been less when the code was introduced) but also the drag on motivation. It’s a huge gumption trap to be making steady progress towards a goal and then be knocked off course by an unnecessary design flaw. The research described in The Progress Principal suggests that small blockers like this have a disproportionate impact compared to their size. Time to break for a cup of tea.

I believe that software quality is a cumulative property. It’s the accumulation of many small good or bad design decisions that either make a codebase productive to work with or just too expensive to maintain.

…and, right on cue, Rangwald talks about The Tyranny of the Urgent.

1) The details of the domain have been changed to protect the innocent, so please don’t worry too much about the detail.

Thanks to @aparker42 for his comments

If you register with the code free1250, you’ll get a discount of 1250 DKK and Goto will donate the same amount to Computers for Charities

Some of us are then rushing to Goto Amsterdam, where I’ll be giving the talk again on Friday. Again the code free1250 will do something wonderful, but I’m not quite sure what.

I think there’s a description in Paul Adams’ talk about online vs. offline social networks of how Dependency Injection goes bad, particularly when using one of the many automated frameworks.

Adams describes a research subject Debbie who, in “real life” has friends and contacts from very different walks of life. She has friends from college with alternative lifestyles who post images from their favourite LA gay bars. She also trains local 10-year olds in competitive swimming. Both the college friends and swimming kids have “friended” her. She was horrified to discover that these two worlds had inadvertently become linked though her social networking account.

This is the “Facebook problem”. The assumption that all relationships are equivalent was good enough for college dorms but doesn’t really scale to the rest of the world, hence Google+. As Adams points out,

Facebook itself is not the problem here. The problem here is that these are different parts of Debbie’s life that would never have been exposed to each other offline were linked online.

Like most users, Debbie wasn’t thinking of the bigger picture when she bound the whole of her life together. She was just connecting to people she knew and commenting on some pictures of guys with cute buns.

Simile alert!

Let’s revisit the right-hand side of that illustration.

This is Nat‘s diagram for the Ports and Adapters pattern. It illustrates how some people (including us) think system components should be built, with the domain logic in the centre protected from the accidental complexity of the outside world by a layer of adapters. I do not want to have my web code inadvertently linked directly to my persistence code (or even connected to LA gay bars).

That’s the trouble with the use of DI frameworks in systems that I’ve seen, there’s only one level of relationship: get me an object from the container. When I’m adding a feature, I just want to get hold of some component—and here’s an easy way to do it. It takes a lot of rigour to step back at every access to consider whether I’m introducing a subtle link between components that really shouldn’t know about each other.

I know that most of the frameworks support managing different contexts but it seems that, frankly, that’s more thinking and organisation than most teams have time for at the beginning of a project. As for cleaning up after the fact, well it’s a good way to make a living if the company can afford it and you like solving complex puzzles. More critical, however, is that the Ports and Adapters structure is recursive. Trying to manage the environments of multiple levels of subsystem with most current containers would be, in Keith Braithwaite‘s words, “impossible and/or insane”.

new again

The answer, I believe, is to save the DI frameworks for the real boundaries of the system, the parts which might change from installation to installation. Otherwise, I gather object assembly into specialised areas of the code where I can build up the run-time structure of the system with the deft use of constructors and new. It’ll look a bit complex but no worse than the equivalent DI structure (and everyone should learn to read code that looks like lisp).

If I later find that I can’t get access to some component that I think I need, that’s not necessarily a bad thing. It’s telling me that I’m introducing a new dependency and sometimes that’s a hint that a component is in the wrong place, or that I’m trying to use it from the wrong place. The coding bump is a design feedback mechanism that I miss when I can just pull objects out of a container. If I do a good job, I should find that, most of the time, I have just the right components at the time that I need them.

I think this fits into Dave Snowden’s Cynefin framework, where he distinguishes between the ordered and unordered domains. In the ordered domain, causes lead to effects. This might be difficult to see and require an expert to interpret, but essentially we expect to see the same results when we repeat an action. In the complex, unordered domain, there is no such promise. For example, we know that flocking birds are driven by three simple rules but we can’t predict exactly where a flock will go next. Groups of people are even more complex, as conscious individuals can change the structure of a system whilst being part of it. We need different techniques for working with ordered and unordered systems, as anyone who’s tried to impose order on a gang of unruly programmers will know.

Loosely, we use rules and expert knowledge for ordered systems, the appropriate actions can be decided from outside the system. Much of the software we’re commissioned to build is about lowering the cost of expertise by encoding human decision-making. This works for, say ticket processing, but is problematic for complex domains where the result of an action is literally unknowable. There, the best we can do to influence a system is to try probing it and be prepared to respond quickly to whatever happens. Joseph Pelrine uses the example of a house party—a good host knows when to introduce people, when to top up the drinks, and when to rescue someone from that awful bore from IT. A party where everyone is instructed to re-enact all the moves from last time is unlikely to be equally successful¹. Online start-ups are another example of operating in a complex environment: the Internet. Nobody really knows what all those people will do, so the best option is to act, to ship something, and then respond as the behaviour becomes clearer.

Snowden distinguishes between “fail-safe” and “safe-fail” initiatives. We use use fail-safe techniques for ordered systems because we know what’s supposed to happen and it’s more effective to get things right—we want a build system that just works. We use safe-fail techniques for unordered systems because the best we can do is to try different actions, none of which is large enough to damage the system, until we find something that takes us in the right direction—with a room full of excitable children we might try playing a video to see if it calms them down.

At the technical level, Test-Driven Development is largely fail-safe. It allows us, amongst other benefits, to develop code that just works (for multiple meanings of “work”). We take a little extra time around the writing of the code, which more than pays back within the larger development cycle. At higher levels, TDD can support safe-fail development because it lowers the cost of changing our mind later. This allows us to take an interim decision now about which small feature to implement next or which design to choose. We can afford to revisit it later when we’ve seen the result without crashing the whole project.

Continuous deployment environments such as at Forward², on the other hand, emphasize “safe-fail”. The system is partitioned up so that no individual change can damage it, and the feedback loop is tight enough that the team can detect and respond to changes very quickly. That said, even the niftiest lean start-up will have fail-safe elements too, a sustained network failure or a data breach could be the end of the company. Start-ups that fail to understand this end up teetering on the edge of disaster.

We’ve learned a lot over the last ten years about how to tune our development practices. Test-Driven Development is no more “over” than Object-Orientation is, it’s just that we understand better how to apply it. I think our early understanding was coloured by the fact that the original eXtreme Programming project, C3, was payroll, an ordered system; I don’t want my pay cheque worked out by trying some numbers and seeing who complains³. We learned to Embrace Change, that it’s a sign of a healthy development environment rather than a problem. As we’ve expanded into less predictable domains, we’re also learning to Embrace Failure.

I’ll also be presenting an engaging rant on why we should aspire to living and working in a world where stuff just works.

If you quote the promotion code FREE100 when you sign up, QCon will give you a discount of £100 and the same amount to Crisis Charity.

participants for whom an elderly stereotype was primed walked more slowly down the hallway when leaving the experiment than did control participants, consistent with the content of that stereotype.

In the “Bang” experiment, they took two queues of people entering the Science Museum and placed pictures of the elderly and infirm around one queue, and the young and active around the other. The result was the same, people in the queue with the elderly images took significantly longer to walk into the building.

It’s striking that such a small thing can affect how we behave.

Now, look around your work environment and consider what it’s priming you for. Are you seeing artefacts of purpose and effectiveness? Or does it speak of regimentation and decay? Now look at your computer screen. Are you seeing an environment that emphasises productivity and quality? Or does it speak of control and ugliness?

It’s amazing that some of us get anything done at all.

This isn’t about spending lots of money to look nice (although that espresso machine is appreciated). I suspect that the sort of “funky, creative” offices that get commissioned from designers dressed in black are usually an upmarket version of motivational posters.

My guess is that a truly productive environment must have some “authenticity” for the people who spend most of their days in it. Most geeks I know would be happy with a trestle-table provided they get to spend the difference on a good chair and powerful kit, and other disciplines might have other priorities.

But then, perhaps every environment is authentic since the organisation is making clear what it really values most. And what might that imply?…

This is all Chris Matts‘ idea. He realised that the problem with the “Technical Debt” metaphor is that for managers debt can be a good thing. Executives can be required to take on more debt because it makes the finances work better, it might even be encouraged by tax breaks. This is not the same debt as your personal credit card. Chris came up with a better metaphor, the Call Option.

I “write” a Call Option when I sell someone the right, but not the obligation, to buy in the future an agreed quantity of something at an price that is fixed now. So, for a payment now, I agree to sell you 10,000 chocolate santas¹ at 56 pence each, at any time up to 10th December. You’re prepared to pay the premium because you want to know that you’ll have santas in your stores at a price you can sell.

From my side, if the price of the santas stays low, I get to keep your payment and I’m ahead. But, I also run the risk of having to provide these santas when the price has rocketed to 72 pence. I can protect myself by making arrangements with another party to acquire them at 56 pence or less, or by actually having them in stock. Or, I can take a chance and just collect the premium. This is called an unhedged, or “Naked”, Call. In the financial world this is risky because it has unlimited downside, I have to supply the santas whatever they cost me to provide.

Call options are a better model than debt for cruddy code (without tests) because they capture the unpredictability of what we do. If I slap in an a feature without cleaning up then I get the benefit immediately, I collect the premium. If I never see that code again, then I’m ahead and, in retrospect, it would have been foolish to have spent time cleaning it up.

On the other hand, if a radical new feature comes in that I have to do, all those quick fixes suddenly become very expensive to work with. Examples I’ve seen are a big new client that requires a port to a different platform, or a new regulatory requirement that needs a new report. I get equivalent problems if there’s a failure I have to interpret and fix just before a deadline, or the team members turn over completely and no-one remembers the tacit knowledge that helps the code make sense. The market has moved away from where I thought it was going to be and my option has been called.

Even if it is more expensive to do things cleanly (and I’m not convinced of that beyond a two-week horizon), it’s also less risky. A messy system is full of unhedged calls, each of which can cost an unpredictable amount should they ever be exercised. We’ve all seen what this can do in the financial markets, and the scary thing is that failure, if it comes, can be sudden—everything is fine until it isn’t. I’ve seen a few systems which are just too hard to change to keep up with the competition and the owners are in real trouble.

So that makes refactoring like buying an option too. I pay a premium now so that I have more choices about where I might take the code later. This is a mundane and obvious activity in many aspects of business—although not, it seems, software development. I don’t need to spend this money if I know exactly what will happen, if I have perfect knowledge of the relevant parts of the future, but I don’t recall when I last saw this happen.

So, the next time you have to deal with implausible delivery dates, don’t talk about Technical Debt. Debt is predictable and can be managed, it’s just another tool. Try talking about an Unhedged Call. Now all we need is a way to price Code Smells.

1) There is an apocryphal story about a trader buying chocolate santa futures and forgetting to sell them on. Eventually a truckload turned up at the Wall Street headquarters.

— Nicolo Machiavelli, The Prince

a complete engineer can code for latency or throughput and knows when and how to switch

What I think he actually said is that we should make engineering trade-offs. When we’re concerned with learning, then we want the fastest turnaround possible. It’s like a physics apparatus, it’s over-engineered if it lasts beyond the experiment. But, if we’re delivering stuff that people will actually use, that we want them to rely on, then the trade-off is different and we should do all that testing, refactoring, and so on that he’s been talking about for the last ten years. Kent brushes over that engineering stuff in his talk, but it’s easy to forget how rare timely, quality delivery is in the field.

My favourite part is Kent’s answer to the last question. A stressed manager or owner asks how to get his developers to stop being so careful and just ship something. Kent’s reply is to present the developers with the real problem, not the manager’s solution, and let them find a way. What the manager really wants is cheap feedback on some different options. The developers, given a chance, might find a better solution altogether, without being forced into arbitrarily dropping quality.

Good developers insist on maintaining quality, partly to maintain pride in their work (as Deming proposed), but also because we’ve all learned that it’s the best route to sustained delivery.

As Brian Marick pointed out recently, it’s about achieving more (much more) through relationships, not one side or another achieving dominance.

via Gemba Panta Rei

Twain also wrote of opera, “that sort of intense but incoherent noise which always so reminds me of the time the orphan asylum burned down.”

We happen to be using the Spring database helper classes to talk to Oracle with stored procedures. It turns out that there’s a bug in the driver that means that you have to jump through a few hoops to pass values in when the input parameter type is a table. This should be equivalent to an array, but apparently it isn’t, so you have to set up the callable statement correctly. Where to do this was not obvious (to us) in the Spring framework.

Here’s an example stored procedure declaration:

CREATE TYPE VARCHARTAB IS TABLE OF VARCHAR2(255);

CREATE PACKAGE db_package {
  TYPE list_type IS TABLE OF VARCHAR2(50) INDEX BY BINARY_INTEGER;
PROCEDURE a_stored_procedure(
  table_in IN list_type
)

The table_in parameter type list_type is declared within a package, which means we can’t declare the parameter as an OracleTypes.ARRAY when setting up the statement. Instead we declare it as the type of the table contents OracleTypes.VARCHAR

class MyProcedure extends StoredProcedure {
  public MyProcedure(DataSource dataSource) {
    super(dataSource, "db_package.a_stored_procedure");
    declareParameter(new SqlParameter("table_in", 
                                      OracleTypes.VARCHAR));
    compile();
  }
  
  void call(String... values) {
    execute(withParameters(values));
  }

Here’s the money quote. When setting up the parameter, you need to provide it with a SqlTypeValue. Don’t use one of the helper base classes that come out of the box, but create an implementation directly. That gives you access to the statement, which you can cast and set appropriately.

   private Map<String, Object> withParameters(String... values) {
      return ImmutableMap.of("table_in",
                             oracleIndexTableWith(50, values));
   }

   private  <T> SqlTypeValue 
   oracleIndexTableWith(final int elemMaxLen, final T... values) {
     return new SqlTypeValue() {
       @Override
       public void setTypeValue(
         PreparedStatement statement, int paramIndex, 
         int sqlType, String typeName) throws SQLException 
      {
         ((OracleCallableStatement)statement).setPlsqlIndexTable(
            paramIndex, values, values.length, values.length,  
            sqlType, elemMaxLen);
       }
     };
   }
}

That’s it. Happy copy and paste.

He has a point of contention in that he’s doubtful about the value of our end-to-end tests. To be more precise, he’s doubtful about the value of our automated end-to-end tests, a view shared by J.B.Rainsberger, and Arlo Belshee and Jim Shore. That’s a pretty serious group. I think the answer, as always, is “it depends”.

There are real advantages to writing automated end-to-end tests. As Nat pointed out in an extended message to the mailing list for the book,

Most significantly to me, however, is the difference between “testing” end-to-end or through the GUI and “test-driving”. A lot of people who are evangelical about TDD for coding do not use end-to-end tests for driving design at the system scale. I have found that writing tests gives useful design feedback, no matter what the scale.

For example, during Arlo and Jim’s session, I was struck by how many of the “failure stories” described situations where the acceptance tests were actually doing their job: revealing problems (such as deployment difficulties) that needed to be fixed.

Automating an end-to-end test helps me think more carefully about what exactly I care about in the next feature. Automating tests for many features encourages me to work out a language to describe them, which clarifies how I describe the system and makes new features easier to test.

And then there’s scale. Pretty much anything will work for a small system (although Alan Shalloway has a story about how even a quick demonstrator project can get out of hand). For larger systems, things get complicated, people come and go, and the team isn’t quite as confident as it needs to be about where things are connected. Perhaps these are symptoms of weaknesses in the team culture, but it seems wasteful to me to take the design experience we gained while writing the features not encode it somewhere.

Of course this comes at a price. Effective end-to-end tests take skill, experience, and (most important) commitment. Not every system I’ve seen has been programmed by people who are as rigorous as Nat about making the test code expressive or allowing testing to drive the design. Worse, a large collection of badly written end-to-end tests (a pattern I’ve seen a few times) is a huge drag on development. Is that price worth paying? It (ahem) depends, and part of the skill is in finding the right places to test.

So, let me turn Brian’s final question around. What would it take to make automated end-to-end tests less scary?

There are still a few places left for the London XpDay, an event designed by practitioners for practitioners.

We’re trying the half-Open Space format again, with a day of prepared sessions (some promising experience reports this year) leading to a day of ad-hoc sessions. This means we can have a conference that’s more responsive to the needs of the attendees in the room—if I want to cover a topic I can propose a session.

And we have some interesting keynotes. Apart from Mark Striebeck, talking about scaling up some agile techniques as only Google can, we’re continuing our tradition of bringing in ”outside“ speakers to trigger discussion. We have Doron Swade (who built the calculating engine in the Science Museum) talking about Babbage, and storyteller Terry Saunders.

Nat and I will also be using the opportunity to launch our book in the UK.