Chuck Connell is a software consultant and writer in Bedford, MA. He can be reached at www.BeautifulSoftware.com.

What is software? What is it about software that takes so long to create? And why does software development so often go wrong, compared to other kinds of engineering?

Along with my day job as a software consultant, I like to think about the essence of this stuff I work with every day. I wonder why it is different from other "stuff" that humans build things with, such as bricks and steel and chemicals.

Computer scientists have proposed many ways of looking at software, including as a function with inputs and an output, and as instructions for transforming computer memory from one state to another. I like to think of software, however, as a machine. It is a machine we cannot touch, but it is a machine nonetheless. (I am not the first to suggest this view.) Equivalently, we can think of software as one part of a machine that includes the computer hardware, allowing us to see software as a component of a physical machine.

When we create a new software system we are creating a new machine. The principles that apply to good machine design also apply to good software design, such as durability, maintainability, and simplicity. But, if this is true, why does software development seem harder than mechanical or structural engineering? After all, people build airplanes, bridges, skyscrapers and factories, on time and on budget. (Not always of course, but often.) It is rare that large physical engineering projects are simply abandoned half-built, after years of effort, with millions of dollars spent, left to rust under the elements. But there are many examples of this scale of failure for software projects, including the Denver Airport, FBI's computer upgrade, and the IRS's modernization effort. Many, many others were never made public because their failures were hidden by organizations not wishing to be embarrassed by the scope of their incompetence.

So what, specifically, is it about software that causes large software projects to go wrong more often than with other kinds of engineering? This question has been examined before, including in Dreaming in Code, Patterns in Failure, and Why Software Is So Bad. Some of the proposed answers are poor requirements documentation, and that software is just plain harder than other kinds of engineering. I believe there is another answer, however, which has not been stated clearly.

The problem is not the nature of software; software is just a type of machine and is amenable to known methods for good machine design and project management. The problem is the way we approach software projects and our expectations for their outcomes. In short, we expect too much. Too often we try to "invent the world" with a new software project, instead of relying on well-known designs and methods that are likely to succeed.

Consider this fictional mechanical engineering project that is run like many software projects.

The motivation for this project is that cars are a very poor form of transportation for individual people. This has been widely recognized for a long time. We want a smaller, lighter, cleaner, less expensive device for personal, local transportation. We will call the new invention "Personal Transportation Device 1000", stating its intended selling price in US Dollars. For individual commuting and errands within 50 miles, we want PTD-1000 to make the current automobile obsolete. We do not want the device to use existing, already congested, roads so PTD-1000 will fly. Our goals include low fuel cost and no pollution, so the motor will be powered by helium fusion. Fusion is an emerging standard, but we believe that this project will provide synergy to fusion research, both driving the research and serving as a test bed for it. We want the device to be light, which will contribute to efficiency and allow the single user to pick it up, so we will construct PTD-1000 primarily from Rearden Metal. The design team recognizes that the formulation for this metal is not yet finished, so we will assign our A-team of engineers to finish this work in parallel with the other subsystems. The final key design criterion is that a person who is physically disabled must be able to enter a building after getting there. So PTD-1000 will convert to a wheelchair for use indoors. The participants in this project all understand that it is a substantial undertaking, but enthusiasm is high for the benefits that will be realized at its completion. There is buy-in by all stakeholders. The investors and engineers have committed to a budget and a completion date of Q4 2011. Everyone has agreed to forgo their vacations for the next year in order to meet this schedule.

Of course, this fictional invention is absurd. Anyone with common sense can see that it will fail dramatically. But take the above scenario, and substitute software concepts for the physical details, and you have an accurate description of many real software projects.