Is Software Always Late?

Wednesday, 11 September 2002

For a long time, I have been surprised that software is consistently late. It is so consistent, that some simply throw their hands in the air and surrender to it. I have never been happy to accept that because surely the consistency of the problem means that there are likely to be common causes and, thus, common solutions. It is only in the last year I discovered that other people had gone out and done something about this. This article is about agile methods, which look like the best way of tackling some of the big problems in corporate software development.

Just in case you can't digest long articles, I can summarise it with this statement: the client has to be your partner.

Coincident Roles

Have you ever met a programmer that writes a formal functional specification for a personal project? They are more likely to jot down on paper some rough ideas, maybe some design structure to make sure that things can get moving. My recent web site designs are an example of this. I didn't map out the whole experience, I just put together an initial design and kept improving and refining until I was happy.

There is no specification, simply a initial target design. The programmer then refines that design. The important thing here is that the programmer is also the client. The programmer as client is an extremely efficient model. The code and design typically iterates very quickly to the first complete version, or at least to a working prototype.

If we describe this process by separating the two coincident roles, then we have the following:

There is no formal specification, simply an evolving implementation. It's fast. The client is happy to see something and be able to suggest changes on every step. The programmer is happy as there's immediate feedback and a feeling of regular achievement.

It would be nice to think this model would work if the client and programmer were two separate people. However, it is quite common for the formal specification to be considered a necessity in corporate implementations, which acts as an interface between the two. No work occurs unless the specification is complete. Unfortunately, the rigid, formal nature of the specification can cast both client and programmer as victim.

For a small project, drawing up a spec isn't necessarily a problem. For a big project, on the other hand, there are likely to be many dependencies between disparate elements of the project. The client is nervous about signing-off and wants comfort from the developer that the specification isn't missing anything. The developer, on the other hand, refuses to code until final sign-off and is unwilling to take any responsibility for the functional requirements; after all, clients will label anything that doesn't work as a bug, even design flaws - why take on extra responsibility?

Thus, client-vendor conflict is encouraged. Specifications take forever. Once the project enters the client testing phase, it becomes clear a lot of things don't work properly. Interfaces are broken; process flow isn't quite right; users find the system doesn't work quite how they expected. At that point, the developers start looking foolish with statements that would sound amusing in court, "well, that bug isn't a bug", trying to explain to users the difference between a design specification flaw and a coding error. It doesn't really matter. The system doesn't work. Something went wrong.

The Evolution Metaphor

For a long time, software developers have lived with engineering metaphor. Much of the language that developers use is knee-deep in this metaphor. Software engineering. Building software. Code structure. This metaphor suggests that you propose a design in advance and then stick to it. What could be simpler?

It doesn't work. First, your client doesn't know all of the requirements. Second, there are potentially dependencies on other ongoing projects with other teams or software houses. Third, requirements analysis may take so long that the requirements have changed by the time you have finished. Fourth, construction has many repeatable processes and aspects while software development, by nature, is always new. If it wasn't new, then you wouldn't be developing anything, you'd just be selling the last copy of the software you developed.

A better metaphor is evolution (taken from Frederick Brooks' The Mythical Man-Month). Software needs to grow. Each release is merely a cycle of growth. After each cycle, everyone can look at the progress. At every step there is a chance to redirect the course of the development. There is never a point of no return. The concept of the out-of-date specification is replaced with the latest release, always ready and willing to be changed. No-one has to dream up every scenario and cover it in the specification.

No matter how hard you fight the principle that software evolves, it's always there. There is nothing you can do about it. That's why there are always millions of changes after every release. You can't just wish for a "smarter client" who has a full grasp of what a developer needs to know. Clients are people. And people can't plan the universe in advance.

I find it interesting that an evolutionary approach is a mode of development that every programmer is already used to, but despite this, we become conditioned into demanding a specification from a client. We couldn't produce specifications for our own projects, so what the heck makes us think that a client can do it for theirs?

Agile Methods

Let's not get carried away. Just because we dispense with a formal specification, doesn't mean we just hack away without any thought for design, milestones or responsibility. This is where the agile methodologies come in. They present an approach of rapidly developing software, but sensibly, with care. They are agile not rushed.

Agile methods change the nature of the relationship with the client. The client and the developer can no longer be two separate entities just passing a few e-mails back and forth. They have to work in symbiosis in the evolutionary model. The client no longer expects perfect software to magically appear; the programmer no longer expects the client to explain the full requirements. Together, they build the software piece by piece. Sometimes pieces have to be removed or rewritten for the whole project to go forward, but this is the nature of the game now. The software development is agile and responds when change is necessary.

An important aspect of agile software development is that you need to have confidence that you can change the code safely. Extreme Programming, for example, underpins everything with automated testing, so you can see before each daily release whether the system is okay or not. Similarly, Microsoft produce releases daily and then run automated tests to see ensure that functionality that worked before still works. The Windows NT test apparently took all night but, despite that, was still considered an key element of the development process.

I want to rant on and on about the importance of automated testing. It's just so crucial. Developers often show great fear when it comes to changing code that's been through user testing, because the confidence they have in the code will be destroyed. By automating the tests, you have confidence forever. You can keep changing that code, again and again, and if you break anything, you know immediately. This is one thing that should be an industry-wide standard.

Software developed through an agile approach will gain faster user acceptance and achieve faster convergence with essential user requirements; it is also more likely to meet a production deadline. There are various reasons for that last statement. Whereas the deadline would normally imply the date at which the code must be completed and tested according to the specification, an agile method is more likely to recast a deadline as a point at which the software must be able to benefit the client. Also, the developers are more confident in their code changes and testing is streamlined. Furthermore, it is far easier to trim requirements to meet any deadline when the client can use the system and actually see what changes would bring real benefit.

I have become more and more convinced that agile methods are superior to the heavy-duty specification approach that is standard. It takes a brave development team and client to switch to an agile method, but I think once you have the first project behind you, the next one should be easier.

Regarding Open Source

I should mention Linux and open source projects. Linux is a great operating system because the developer and the client are the same. Linux continues to move forward at a great pace, always improving; it is the OS of choice for system administrators and developers. That's because the system administrators and developers are doing all the developing.

Just look at the GUI side of things though. It's not working out. Linux has done so well behind the scenes, where the client and the developer are one and the same. On the GUI side, you need more ordinary users out there to be able to say what they think makes a nice interface. Some open source developers are unwilling to "dumb down like Microsoft", and also tend to want their claim-to-fame tick box pasted into an application even if it doesn't belong. I know I'm tarring all developers with the same brush, but these are real problems which are bogging Linux and its applications down on the desktop side. Things are further complicated as there are multiple GUIs available.

The biggest problem with open source projects is that the majority effort in open source is conducted by free developers, who are their own client. Perhaps if more corporate bodies are convinced to add contributions to Linux (e.g. Ximian contributes to GNOME) then Linux has a future in the desktop world. Otherwise, it will only be known as the server operating system of choice.

Nothing is Straightforward

Despite my vociferous approval for agile methods, I've not had the opportunity to work on an agile project. Once I do, I am likely to adjust my views; I'm wise enough to know that things aren't as clear cut as I've made out above. If a methodology or approach doesn't feel right, then you shouldn't do it. You can pick and choose aspects you are comfortable with from these methods. No-one is forcing you to do anything (except your manager I guess).

I know that in shrinkwrap development, your client tends to be "a lot of people in the world", which means that change needs to be controlled. You can't just go changing the product according to every e-mail you get sent.

If your client doesn't trust you enough to engage in an agile project then you'll need an approach demanding more metrics and fixed milestones. Trust is essential in the agile method. The client must believe you are working together for a common good and not just treating them like Daddy's credit card by skipping a detailed specification and estimate stage.

Also, if you're a market leader, I guess the last thing a client is expecting when you walk into the room is for you to say, "So, guys, what is it you want?" All those years of experience should have taught you a thing or two about what usually happens, what your clients usually want. If you're the market standard, you should start dictating to the client because it's expected.

If you don't know much about agile methods, go and find out now. A good place to start is Martin Fowler's paper on agile methodologies. Life is too short to keep missing deadlines.