When Bill Gates started to make an impact in the software business, he reportedly said: "I want to be the IBM of software." It looks as though he succeeded all too well - Microsoft looks as bloated and bureaucratic as its giant model, where programming was famously compared to prehistoric monsters wrestling in tar-pits. Where he is breaking with the tradition of his predecessors, IBM's Tom Watsons, Sr and Jr, is that he is getting out early. Last week, Gates announced stage two of a multipart retirement strategy, which started when he stopped being Microsoft's chief executive six years ago. He will still be Microsoft's chairman, but will reduce his day-to-day workload over the next two years.

Watson Sr, who founded IBM, clung on tenaciously into his 80s, and many thought Gates would do the same. But for more than a decade, he has had another interest: trying to improve the lot of the world's poorest. He started devoting his fortune to good causes, now done through the Bill & Melinda Gates Foundation. Gates's plan is to reverse the balance of the time he devotes to Microsoft and his charitable works.

At the hastily arranged press conference to announce the change, Gates said: "In the early days, I liked to review every line of code, to interview every job applicant. Today, our products include millions of lines of code, we hire thousands of people a year, I've had to lighten up in both of those areas."

Clearly he hasn't been reviewing the code for Vista, the next version of Windows, which has exhibited all the tar-pit tendencies described in Fred Brooks's classic book on programming, The Mythical Man-Month. Brooks was writing about the development of the IBM 360 mainframe family, launched in 1964, which was by far the largest private project of the age. Vista is on a similar scale, and in a blog post called Broken Windows Theory, Philip Su describes it as "the largest concerted software project in human history. The types of software management issues being dealt with by Windows leaders are hard problems, problems that no other company has solved successfully. The solutions to these challenges are certainly not trivial."

Usual suspects

Having "managed developer teams in Windows for five years", he writes (at www.tinyurl.com/z2wlo) about "the usual suspects" - Windows code is too complicated, and the process has "gone thermonuclear". He also writes about more important cultural effects: there's too much random micro-management, and no one really wants to know if something is going wrong. "Every once in a while, Truth still pipes up in meetings," he writes. "When this happens, more often than not, Truth is simply bent over an authoritative knee and soundly spanked into silence."

No doubt this is true of all large companies. But don't feel too bad, Philip Su: Windows Vista could be doing better than average, by software industry standards. In the US, for example, a Standish Group survey found that more than a third of software projects were simply abandoned unfinished. The average project was 90% over budget and exceeded its schedule by 120%. "In the larger companies, the news is even worse: only 9% of their projects come in on time and on budget," the report said. "And, even when these projects are completed, many are no more than a mere shadow of their original specification requirements. Projects completed by the largest American companies have only approximately 42% of the originally proposed features and functions."

That certainly sounds like Vista - but it was written a decade earlier.

The same problems have affected many British projects, including the Stock Exchange's failed Taurus (Transfer and Automated Registration of Uncertificated Stock), and a significant proportion of UK government IT projects. For a recent example, the NHS's current IT development was budgeted to cost £6.2bn; it will now cost £12.4bn, and could end up at £20bn or more. It's also late. And it's still considered relatively successful.

Either way, the problem isn't getting easier. The Standish Group called its first survey, in 1994, The Chaos Report. By 2001, it was Extreme Chaos. So the real question is: why? We have known since the IBM mainframe software debacle, more than 40 years ago, that very large software projects are very hard to do. Why do we keep making the same mistakes? More to the point, will Microsoft make the same mistake again? Or have we seen the last blockbuster version of Windows?

Viable alternatives

The other thing we know today is that there are viable alternatives, as shown by Apple's development of OS X, by companies such as Google, and the open source movement. Apple had its own struggles in the tar-pits, until it abandoned its operating system development and went out shopping for a replacement. Even after buying NeXT (a little company run by Steve Jobs) for $400m at the end of 1996, and getting the complete Unix-based NextStep operating system, it took more than four years to release OS X 10.0 (Cheetah). Since then, there have been regular "point releases": Puma, Jaguar, Panther and Tiger. If Leopard (10.5) arrives alongside Vista, that will be six releases since 2001 - the year both OS X and Windows XP came out.

To be fair, Microsoft has also released Media Center and Tablet PC versions of Windows XP, plus SP2 and versions of Windows CE and Windows Server. It's clearly more manageable to target useful point releases than embark on a five-year death march to finish something like Vista. Apple even manages to charge for them.

Another idea is to compartmentalise the development and have teams working independently, which is more or less what happens in the open source market. Whether this would be a practical or economically viable option for Windows involves a lot of other issues.

Windows is already a combination of parts from separate teams, and Vista combines multiple developments. Some parts are even released separately, such as Internet Explorer and Windows Media Player. But that could be tricky given the notorious interdependencies that afflict Windows. Also, most copies of Windows are bought by companies who hate "touching the desktop" because it costs them time and money. They like to have one desktop for as long as possible and, if they can, will skip whole versions.

Microsoft has traditionally felt it needed to produce a blockbuster version of Windows just to get companies to upgrade. And pay for another copy. It has tried to get around this by introducing a licensing scheme, Software Assurance, where companies pay a regular fee and get upgrades free. This has not been wildly successful, and it can't be easy to sell a three-year scheme if there are up to five years between free upgrades.

The third option is to run development as a continuous process, essentially what happens with online companies such as Google. This simplifies software development, but again it is hard to sell the idea to companies that want to control and test their own updates. Users can give up control and buy centralised services. But that was one of the planks of Microsoft's .Net strategy - which included an online version of Office - and it didn't take off.

The most likely result is a combination of all these approaches. Microsoft is not going to stop producing things like Windows XP, Windows Server and Microsoft Office, because that is how businesses buy software - and it brings in $1bn a month in profits. But, at the other extreme, Microsoft is spending heavily on web developments to compete with Google, Yahoo and other web-based companies. You can expect to see a very wide range of ideas floated.

Last year, Martin Taylor, Microsoft's general manager of competitive strategy, said the firm was planning to deliver "software as a service", but not using the hosted ASP (application service provider) model it tried with .Net. "We have got a strategy for all of our products of how we deliver value to customers via the cloud. Stage two is how we monetise that value," he said.

In other words, Microsoft will extend its packaged offerings via a mixture of free (supported by advertising) and premium (subscription) online services, such as OneCare and Office Live.

IBM went through a similar transformation. After cheap Intel-based servers appeared in the 1980s, companies were less willing to install giant mainframes, and the company recorded multibillion-dollar losses. But IBM didn't go away - it is still more than twice as big as Microsoft. Nor did mainframes: they still run the country. But IBM changed from a supplier that depended on shipping "big iron" to one selling software and services.

Microsoft has to make the same sort of transition, and it won't be easy. You'd have thought this was just the sort of huge challenge that someone like Bill Gates would find irresistible. Apparently not.

· If you'd like to comment on any aspect of Technology Guardian, send your emails to tech@theguardian.com