I’ve just finished a fortnight’s holiday which I (foolishly) spent mostly in front of a PC developing a never-ending little application. But unexpectedly, despite the trivial nature of my project, I rediscovered a number of important lessons more usually associated with serious application development.

The software I’m writing is a just a little Firefox plugin. I’ve been fiddling with it for so long we’re almost onto the second major release of Firefox since I started, and yet it’s probably not much more than a couple of hundred lines long. You can see it really is a minor enterprise. Despite this it’s been quite a surprise — quite a shock, even — to be reminded of some industrial-strength truths in a small and personal environment.

And they are…

1. Damn, writing software is difficult

What we non-developers know about software we know either by observing or by talking to those who do it. But anyone who wants to be trusted won’t complain about their job or bore you with details they know you don’t want to hear, and so you’ll never hear about everything there is to know about software development, even when you ask.

So one thing I discovered — again — is that writing software is really difficult. Sometimes I was flying, but more often I was crawling: piecing together information from different sources, trying to understand what was possible, learning all kinds of technologies (Javascript, XUL, XPCOM), trying use them well, but more often trying to get them to work at all.

It’s made me respect all the more the people I work with every day who make it look so easy. Every line you write is pure logic and needs to work 100%. This is not like writing a letter or a sales proposal, where a sentence that’s only 95% perfect is more than adequate. This is like writing a legal contract from scratch for a particularly unpredictable world. Every line needs to be watertight.

2. Simple design really is difficult

Simple design is another way of saying that the software should be easy to work with and modify. You’d think this would be hard to get wrong, particularly with something which is being built as new: surely you just add small, simple pieces one at a time, with each one adding the next feature on the list. What could go wrong?

I found out first hand that simple design is difficult to achieve, even in very simple scenarios. For example, my application has a class which handles the user’s preferences. It needs to be initialised for when the application starts, and it needs to take and save the preferences when requested. All this takes place within a single object. So I wrote an initialisation function which set up the initial preferences according to how it’s been configured, I tested it, and all was well. Then I added some functions to take updated values and save them, tested those, and all continued to be well. Total: about 20 lines of code over four functions.

Then I ran the whole in an integrated environment and weird errors started occurring. It took me about two hours to figure out what was going on: it was a strange combination of unexpected start-up values, Firefox calling the initialisation function at unexpected times, and some confusing logic of my own which was supposed to protect double-initialisation. In the end I decided the best solution was to throw out the whole idea of initialisation. Now the object just takes and save preferences, and you can do that when the application starts if you want. Total: 5 lines.

The point of this is that an apparently simple and obvious design was actually too complicated to sustain. I’m very pleased that the solution was to delete lines and simplify. But I could only do this so easily because I had complete control of the code. In other circumstances there might have been other systems which were relying on that initialisation code (however flawed it may have been), and I might have had to add to the existing complications to solve my problem; or I might not have had the time to take a fresh look at the code and simply built around the flaws out of a sense of fear of touching the wrong thing.

This is a tiny example from a tiny piece of work, but it showed me how easy it is to go wrong with a design, and how easy it is to produce software that is complicated, hard to understand, and time-consuming to fix and evolve.

3. Learning a language is more about culture than syntax

I sometimes get CVs from people who claim to know about 10 programming languages, and I’m always doubtful. Just because you’ve written an application in a language it does not follow you’ve done it well. Knowing the syntax is only the first step. You also need to have good knowledge of any libraries, and finally you need to know how to work with the grain of a language. This means you’ll use different idioms, and structure your solutions in different ways.

In my own case I’ve been writing Javascript, but it stinks because I’ve tried to use it like Java. I’ve been stuck in my old Java ways, like creating classes and carving out a deeply-nested namespace. It’s Javascript, Jim, but not as we know it. It works, it makes sense, but it looks clunky and… well, it just feels wrong, dammit.

Javascript is a prototyping language. I can even tell you what that means, but only with my head, not with my heart. I use the prototyping as a hoop to jump through to get it to do the Java-y things that I know I shouldn’t be doing in the first place. Being a prototyping language doesn’t mean Javascript is a second-class language, or a dumbed-down Java. It means it’s a different kind of language to Java and should be treated as a first class language with its own ways of getting things done.

It’s a cultural thing, and you can’t claim to really know the language if you don’t operate comfortably in its culture. I don’t really know Javascript.

4. How did I ever live without automated tests?

Possibly by not spending my holidays sat in front of a PC. But aside from that, I continue to wonder at the marvel that is automated testing, and unit testing in particular. To be able to implement a change and not have to trouble your brain about the consequences is very liberating, allowing you to move ahead with confidence. It does take some work to set up the environment, but the results are worth it.

That said…

5. Your automated tests won’t cover everything

In one of my functions I unexpectedly found a truck-sized hole which had gone undiscovered despite seemingly comprehensive automated test coverage. (A loop which had a “break” when it should have had a “continue”, meaning great swathes of actions got skipped in most circumstances.) I only discovered this through integration testing (which is the fancy name I’m using for what was really “trying it out”), and found that a quirk in my unit testing setup had caused the mistake to be missed. Once I had found the cause I adjusted the main code to be more predictable and put in an automated test to trap the error, but it was only discovered through real hands-on testing.

6. A strong IDE sustains motivation as much as anything

Although I was using Eclipse for development, when working with Javascript it really doesn’t offer the comprehensive support you get with Java. Because Javascript is a dynamic language, and no doubt also because of the state of JSEclipse, there’s very weak support for code completion, refactoring and so on.

The consequence of this was a loss of code-writing speed, but much more than that I was suddenly able to see how easily a weak IDE allows bad software to be produced. There were many occasions when I knew that I should tidy up or refactor something, but was then suddenly hit with a premonition of the tedious steps I’d have to go through: working out which files to pick on, the manual search-and-replace, checking the context before I made a change. I had to force myself to get on with the tidying up despite knowing how painful it would be, focusing on the long-term results, and safe in the knowledge that for this little personal project I didn’t have a deadline.

It became clear to me that so many people must come under a constant barrage of pressure, with only their current strength of character to defend against the pressures of deadlines and short-term wins. It’s inevitable that too often they will give in to those pressures, leaving cumbersome code building up, and ultimately gumming up the works of the system. A strong IDE removes the barriers to those virtuous tasks of improving design and allows you not only to do your job, but to do it well.

7. Estimation is difficult

Which is just an excuse for the shameful truth: all my estimations were out by a factor of four. This is embarrassing because it’s not as if I’ve never written software before.

In retrospect the mistake I was making was to look at the component parts of a task, guess how long they’d each take, and add them up. What I should have done was try to take the forthcoming work, identify similar previous tasks, and from past experience see how long they actually took.

At the end of the day experience is good, but it’s how you use it that really counts.

8. You can’t know all your requirements up front

This is familiar to anyone who’s bought into Agile, but it hits home hard when it’s you who’s the user setting the requirements.

I’ve written countless requirements specifications in the past (in 60-page documents, on task cards, wherever) — I thought I really ought to be clear-thinking enough to know my own requirements up front. Wrong again. As the UI came together, as one idea sparked off another, and as I had chances to step away from the code to think about things from a distance, I started to see that my feature set was really rather disjoint — almost random. These were moments of clarity that on the one hand caused me to add requirements, but in doing so I was recasting the software with a new perspective. I was starting to see what the software should be doing, which was not quite the same as what I’d started out on.

Release date

Fortunately I’ve not sent out a press release announcing a release date, held a press conference, or hired the London Eye for a glitzy media event. I’m just writing software for fun, and at this rate it’s probably not going to ever see the light of day. But even then, it’s been startling to find that the germs of some of the Big Ideas of software development are still present in the smallest of projects.