Ivar Jacobson is a father of components and component architecture, use cases, aspect-oriented software development, modern business engineering, the Unified Modeling Language and the Rational Unified Process. Ian Spence is Chief Scientist at Ivar Jacobson International where he specializes in the agile application of the Unified Process. They can be contacted at [email protected]

Bertrand Meyer and I recently presented an article entitled Methods Need Theory which outlined why we believe that methods need theory. In this article, Ian Spence and I expand on the foundation provided by the previous article to examine more detail at why we will all benefit from establishing such an underlying theory.

--IJ

Our Greatest Challenge: Understanding How To Build Great Software

Do we really know how to develop great software? The answer for many people is clearly yes. But do we know how to communicate and continuously improve the way that we develop software? Do we really understand the best way to communicate and share our knowledge? The answer, as we saw in the previous article is clearly no!

Do we stand on quicksand or the shoulders of giants?

Have you ever taken the time to investigate a new method or practice only to find that it is just the re-branding and regurgitation of ideas that you have seen many times before?

Have you ever got frustrated that every new idea about software development seems to be at the expense and in aggressive competition with everything that has gone before?

Does it seem to you that following that latest software development trend has become more important than producing great software?

Have you noticed how in their hurry to forge ahead people seem to throw away the good with the bad? Instead of learning from their own experience and building on top of the good things that they do, they heedlessly throw everything away and start again with something they believe is new. It is as though they have no solid knowledge to stand upon.

This behavior can be seen in the way that many teams carelessly discard expensive process and tool investments, almost before they have even tried them. Every project adopts a new approach. Every time someone changes their job they have to learn a new approach before they can get on with the real task at hand. This is not effective; people cannot learn from experience as they are forever starting over.

The bottom line is that nothing new ever becomes properly established -- even today after a number of "modern" software development trends the most popular approaches to software development are prescriptive waterfall development or uncontrolled hacking. As an industry we cannot get anything to stick and nothing really seems to change.

But now we're all Agile surely all our problems are solved?

The latest trend to sweep the industry is "being agile".

Now let's be quite clear here, the "agile" movement has made a very positive contribution to the software industry. (If you have seen any of our recent presentations at conferences around the world you know that we are both big fans of Agile and supporters of the Agile Manifesto.) It has reminded us that people matter first and foremost when developing software. This is not really new but it is important, and it did seem to have been neglected by the previous, more technically oriented, fashions such as object orientation, and programming in Java. By presenting a set of values the agile manifesto created something robust and resilient that can withstand the waves of change that will be bought on by the next trend. (A new trend is as sure to follow the agile trend as night is to follow day.)

It's a shame that the same cannot be said for the many agile methods that have promoted themselves as supporting the agile philosophy. For a movement that values people over process and tools it has certainly given us a lot of "new" processes and tools. The majority of these have been effective by putting the team back at the heart of what is done to develop software but in bringing this back into focus, much has been lost or obscured as new terms are introduced for old things, creating the illusion of something completely new.

This constant re-packaging and re-branding of old ideas produces a constant churn in the way that software development teams work. Rather than re-focusing peoples' efforts away from wasted work onto the production of quality software all that seems to happen is an arbitrary renaming of the things they do and produce.

Even with something as correct and beneficial as the agile philosophy the underlying message can get lost in the churn and hype. We are already starting to see a backlash against agile and our fear is that the benefits will be lost as early adopters move onto the next bandwagon and the late majority re-assert their right not to change to something that has obviously gone out of fashion.

All that seems to happen is that we add more buzz-words and conflicting terminology, and we are all becoming deadly tired of all the buzz!

Addressing the challenge: developing a basic theory describing what software engineering actually is

It is clear that we need to stop chasing after fads and easy answers that forever disappoint, and that we need to do it without discouraging innovation and the generation of new ideas. People need to stop constantly re-packaging and re-branding old ideas just for the sake of it. Instead they should focus on helping people understand how to build great software. But how can we facilitate this change in emphasis? In our opinion, this theory is right in front of us -- we just need to grab it.

First we should start with all the popular methods, processes, and practices out there and find the "truth" of software engineering. We can then describe and capture this small set of essential concepts in the form of a minimal, practice independent process -- we call this small set of essential things our "kernel". Then using this kernel as our starting point we can analyze existing processes and methods and identify the practices that they contain. By starting from the kernel we can describe the practices in a way that allows them to be compared and composed.

Now this way of producing our theory is not in its self a theory. This is something that we have already done. By studying a number of methods, including XP, Scrum, and the Unified Process our team has identified a kernel with some 20+ elements, things we always do or always produce. On the surface, there may appear to be large differences in these studied methods and the ways we work with them but under the covers they share the same DNA. As an example, you can capture requirements with features or with use cases or with user stories, and you can use these requirements without a lifecycle, with the Unified Process lifecycle or even (as some people still insist upon) a waterfall lifecycle. There is definitely a common basis for the many methods, which can be captured in the form of a small, simple set of kernel elements.

Now we are not presumptuous enough to propose that our kernel provides the needed theory. More and greater minds than ours are needed to do this. But we would present it as evidence of what can be done and that the theory we need is within reach.