Steve Freeman Rotating Header Image

Machiavelli on code quality

As the doctors say of a wasting disease, to start with, it is easy to cure but difficult to diagnose. After a time, unless it has been diagnosed and treated at the outset, it becomes easy to diagnose but difficult to cure.

— Nicolo Machiavelli, The Prince

via Dee Hock, Birth of the Chaordic Age

Not a charter for hackers

Just had to turn off comments since this post has become a spam target. Sorry.

Update: Kent since tweeted this nice one-liner:
a complete engineer can code for latency or throughput and knows when and how to switch


Kent Beck’s excellent keynote at the Startup Lessons Learned Conference has been attracting some attention on The Interweb. In particular, it seems like he’s now saying that careful engineering is wasteful—just copy and tweak those files to get a result now. I can already hear how this will be cited by frustrated proprietors and managers around the world (more on this in a moment).

What I think he actually said is that we should make engineering trade-offs. When we’re concerned with learning, then we want the fastest turnaround possible. It’s like a physics apparatus, it’s over-engineered if it lasts beyond the experiment. But, if we’re delivering stuff that people will actually use, that we want them to rely on, then the trade-off is different and we should do all that testing, refactoring, and so on that he’s been talking about for the last ten years. Kent brushes over that engineering stuff in his talk, but it’s easy to forget how rare timely, quality delivery is in the field.

My favourite part is Kent’s answer to the last question. A stressed manager or owner asks how to get his developers to stop being so careful and just ship something. Kent’s reply is to present the developers with the real problem, not the manager’s solution, and let them find a way. What the manager really wants is cheap feedback on some different options. The developers, given a chance, might find a better solution altogether, without being forced into arbitrarily dropping quality.

Good developers insist on maintaining quality, partly to maintain pride in their work (as Deming proposed), but also because we’ve all learned that it’s the best route to sustained delivery.

As Brian Marick pointed out recently, it’s about achieving more (much more) through relationships, not one side or another achieving dominance.

Mark Twain again

We should be careful to get out of an experience only the wisdom that is in it—and stop there—lest we be like the cat that sits down on a hot stove-lid. She will never sit down on a hot stove-lid again, and that is well; but also she will never sit down on a cold one any more.

via Gemba Panta Rei

Twain also wrote of opera, “that sort of intense but incoherent noise which always so reminds me of the time the orphan asylum burned down.”

Calling an Oracle stored procedure with a Table parameter with Spring’s StoredProcedure class

I don’t normally do this sort of thing, but this took my colleague Tony Lawrence and me a while to figure out and we didn’t find a good explanation on the web. This will be a very dull posting unless you need to fix this particular problem. Sorry about that.

We happen to be using the Spring database helper classes to talk to Oracle with stored procedures. It turns out that there’s a bug in the driver that means that you have to jump through a few hoops to pass values in when the input parameter type is a table. This should be equivalent to an array, but apparently it isn’t, so you have to set up the callable statement correctly. Where to do this was not obvious (to us) in the Spring framework.

Here’s an example stored procedure declaration:

CREATE TYPE VARCHARTAB IS TABLE OF VARCHAR2(255);

CREATE PACKAGE db_package {
  TYPE list_type IS TABLE OF VARCHAR2(50) INDEX BY BINARY_INTEGER;
PROCEDURE a_stored_procedure(
  table_in IN list_type
)

The table_in parameter type list_type is declared within a package, which means we can’t declare the parameter as an OracleTypes.ARRAY when setting up the statement. Instead we declare it as the type of the table contents OracleTypes.VARCHAR

class MyProcedure extends StoredProcedure {
  public MyProcedure(DataSource dataSource) {
    super(dataSource, "db_package.a_stored_procedure");
    declareParameter(new SqlParameter("table_in", 
                                      OracleTypes.VARCHAR));
    compile();
  }
  
  void call(String... values) {
    execute(withParameters(values));
  }

Here’s the money quote. When setting up the parameter, you need to provide it with a SqlTypeValue. Don’t use one of the helper base classes that come out of the box, but create an implementation directly. That gives you access to the statement, which you can cast and set appropriately.

   private Map<String, Object> withParameters(String... values) {
      return ImmutableMap.of("table_in",
                             oracleIndexTableWith(50, values));
   }

   private  <T> SqlTypeValue 
   oracleIndexTableWith(final int elemMaxLen, final T... values) {
     return new SqlTypeValue() {
       @Override
       public void setTypeValue(
         PreparedStatement statement, int paramIndex, 
         int sqlType, String typeName) throws SQLException 
      {
         ((OracleCallableStatement)statement).setPlsqlIndexTable(
            paramIndex, values, values.length, values.length,  
            sqlType, elemMaxLen);
       }
     };
   }
}

That’s it. Happy copy and paste.

Responding to Brian Marick

Brian’s been paying us the compliment of taking our book seriously and working through our extended example, translating it to Ruby.

He has a point of contention in that he’s doubtful about the value of our end-to-end tests. To be more precise, he’s doubtful about the value of our automated end-to-end tests, a view shared by J.B.Rainsberger, and Arlo Belshee and Jim Shore. That’s a pretty serious group. I think the answer, as always, is “it depends”.

There are real advantages to writing automated end-to-end tests. As Nat pointed out in an extended message to the mailing list for the book,

Most significantly to me, however, is the difference between “testing” end-to-end or through the GUI and “test-driving”. A lot of people who are evangelical about TDD for coding do not use end-to-end tests for driving design at the system scale. I have found that writing tests gives useful design feedback, no matter what the scale.

For example, during Arlo and Jim’s session, I was struck by how many of the “failure stories” described situations where the acceptance tests were actually doing their job: revealing problems (such as deployment difficulties) that needed to be fixed.

Automating an end-to-end test helps me think more carefully about what exactly I care about in the next feature. Automating tests for many features encourages me to work out a language to describe them, which clarifies how I describe the system and makes new features easier to test.

And then there’s scale. Pretty much anything will work for a small system (although Alan Shalloway has a story about how even a quick demonstrator project can get out of hand). For larger systems, things get complicated, people come and go, and the team isn’t quite as confident as it needs to be about where things are connected. Perhaps these are symptoms of weaknesses in the team culture, but it seems wasteful to me to take the design experience we gained while writing the features not encode it somewhere.

Of course this comes at a price. Effective end-to-end tests take skill, experience, and (most important) commitment. Not every system I’ve seen has been programmed by people who are as rigorous as Nat about making the test code expressive or allowing testing to drive the design. Worse, a large collection of badly written end-to-end tests (a pattern I’ve seen a few times) is a huge drag on development. Is that price worth paying? It (ahem) depends, and part of the skill is in finding the right places to test.

So, let me turn Brian’s final question around. What would it take to make automated end-to-end tests less scary?

London XpDay 7th & 8th December

XP Day

There are still a few places left for the London XpDay, an event designed by practitioners for practitioners.

We’re trying the half-Open Space format again, with a day of prepared sessions (some promising experience reports this year) leading to a day of ad-hoc sessions. This means we can have a conference that’s more responsive to the needs of the attendees in the room—if I want to cover a topic I can propose a session.

And we have some interesting keynotes. Apart from Mark Striebeck, talking about scaling up some agile techniques as only Google can, we’re continuing our tradition of bringing in ”outside“ speakers to trigger discussion. We have Doron Swade (who built the calculating engine in the Science Museum) talking about Babbage, and storyteller Terry Saunders.

Nat and I will also be using the opportunity to launch our book in the UK.

Friday 13th, Talking at Skills Matter

Prove you’re not superstitious! I’ll be giving my talk on Sustainable TDD at Skills Matter on Friday, 13th November. Sign up here (if you dare).

This talk is about the qualities we look for in test code that keep the development “habitable.” We want to make sure the tests pull their weight by making them expressive, so that we can tell what’s important when we read them and when they fail, and by making sure they don’t become a maintenance drag themselves. We need to apply as much care and attention to the tests as we do to the production code, although the coding styles may differ. Difficulty in testing might imply that we need to change our test code, but often it’s a hint that our design ideas are wrong and that we ought to change the production code. In practice, these qualities are all related to and support each other. Test-driven development combines testing, specification, and design into one holistic activity.

I just ran it at the BBC and people seemed to like it.

If you miss this opportunity, you can always see it at QCon San Francisco.

QCon San Francisco

QCon

I’m running a track at QCon in San Francisco on Friday 20th November. The topic is Technical Skills for Agile Development, and it’s about some of the technical essentials that Agile teams need to keep moving.

I’ll be presenting a session, based on material from our book, on how to live with your tests over the long term.

See you there?

Test-Driven Development is not an elite technique.

This “Darwinian” post that TDD Is Not For the Weak says that not everyone can cope with TDD, it’s for “the Alpha, the strong, the experienced”. I don’t want to believe this, because I think that developers who can’t cope with any level of TDD shouldn’t be coding at all, so I won’t.

This claim has been made for every technical innovation I’ve seen so far (objects, event-driven programming, etc, etc). Sometimes it’s true, but most of the time it’s about what people are used to. Michael Feathers pointed out a while ago that the Ruby community is happily exploiting techniques such as meta-programming that were traditionally regarded as needing a safe pair of hands. What’s changed is that a generation has grown up with meta-programming and doesn’t regard it as problematic. Of course, there will be a degradation in understanding as an idea rolls out beyond its originators, but there’s still some value that gets through.

Sure there’s a role for people to help the generation that is struggling to pick up a new technique, but that doesn’t mean that TDD itself will always be beyond the range of mortal developers.

A Mugged Liberal

A Liberal is a Conservative Who Hasn’t Been Mugged

I discover that our book Growing Object-Oriented Software, Guided by Tests is already up with the file sharers. This is before Nat and I have even seen a printed copy. I don’t know whether to be flattered that someone thinks our effort is worth pinching or annoyed that the production process has not protected our interests.

If you’ve downloaded a copy, remember that it took us three years to write, which includes a great deal of lost income. The least you can do is tell the world how good it is so that maybe we sell a few more copies.