This post is by a guest blogger.

I am a scientist. I come across a lot of engineers. They are alien life forms, but I have had a symbiotic relationship with them. I found myself getting confused trying to understand the differences between us, until I realised that behaviour depends on environment. You can see how we are different in three ways:

general differences in behaviour,

working on hardware projects,

and working on software projects

The latter is where it gets weird.

General Behaviour

Left to themselves, scientists are obsessed with truth, and engineers worry only about what works. Scientists are impractical purists. Stuff you build — a lab lash up, a sprawling C program — is ragged and messy. It doesn’t matter because you are going to change it soon. You don’t stop until you figure out whats really going on. Engineers are pragmatic. It doesn’t matter whether the theory is actually correct in any deep sense. They worry and puzzle until they get something that works. Then they stop, polish, and admire.

I once listened to an engineer talking about grain silos. He’d done a simulation showing that when you set it up so and so, and jiggle, the big and little bits segregate. My brow furrowed, “Doesn’t that seem to violate the second law of thermodynamics?”

He stared back blankly.

“So?”, he said.

[Answers on a ten pound note please]

Hardware Projects

When it comes to hardware projects the scientist becomes a customer. The purist/pragmatic hats get swapped from the scientist to the engineer.

I am an astronomer. Unlike my condensed matter chums, we don’t potter in the lab. Our work is done with big instruments on even bigger telescopes, that we share as national or international facilities. These things are highly engineered machines and they have to work right. The scientists come up with the concepts – e.g. “what we need for the next step in big cosmological redshift surveys is a multi-fibre spectrograph, only it has to work in the infrared.” Then they work with a team of engineers to design it; then it actually gets built by engineers, technicians, industrial contracts, etc. Normally there is a “Project Scientist” inside the team who makes sure the engineering group delivers what the scientists wanted. So essentially s/he is a kind of commissioning customer.

So does this mean the Project Scientist has to stop the engineers from cutting corners? It’s normally the opposite. The scientist is getting impatient; he wants his new toy. It may be the Scientist’s toy, but its the Engineer’s baby.

“Ahh”, say the engineers, “we need to do more tests.” “Oh, and wait a minute, if we open up the cryostat and use a different type of wire the read noise should go down.” “Well,” they say, “it could pass the CDR we suppose; but there is a 5% chance it will fail at altitude. We should really take it apart and re-align. We should change the baffling while we are at it…”

There comes this point in every Project Scientist’s career where you learn the same lesson that managers learn in industrial teams. Eventually you have to shoot the engineers and ship the product. Yell “enough!” and get that f***er up the mountain.

Software Projects

When getting a piece of kit made, the scientist is no longer a purist. This isn’t her work, it is just something she needs so she can do her work. For the engineers, when building that piece of kit, this is their work, and they are very proud. They are still basically practical folk, but the lure of the technically sweet, and their pride in their work, overcomes their judgement. Now when we come to software projects, the engineers seem to shoot straight past the purist stage and land in a misty realm of philosophy.

I am not talking about normal software development – the code that keeps everyday systems working. Here the engineers are pretty hard headed and practical. I am talking about innovative software projects – difficult datamining tools, ambitious grid infrastructures — roughly the software equivalent of building some new type of spectrograph. Talented software engineers love this stuff. It’s even more fun than transaction software for banks. Already we are in the “oooh, I could make this even better” stage. But it gets worse.

“Well,” they say, “there is whole new technology emerging.” (OWL, REST, Web 3.0, take your pick) “We should change the whole architecture and re-engineer. In fact,” they say, their eyes glistening, “when you think about it, it makes you realise there’s a whole different approach, I mean for users and for us. It’s difficult to explain.” “but its kinda, well. Hmm. I need to think about this.”

Think??!! You’re not supposed to think! You’re supposed to implement! You’re an engineer!

But the scientist is hooked as well. When he wants a piece of hardware, he states what he wants — it needs to do A and B; it needs to fit next to that other gizmo; it mustn’t flex more than X when its tipped. Then he crosses his arms and sticks to his guns.

When he wants a new software tool, he is a bit woolier, but he knows all about this iterative development stuff, and expects he will sharpen his requirements as he sees the prototypes emerge. But he sees the first prototype he says, “oh, now hang on that gives me an idea. Nah, we got this all wrong, what we want is more like blah. In fact, thats really strange, you know, I read this blog the other day, and it made a very interesting point…”

This is why most software projects fail. Not because the software industry is incompetent or immature in its methodologies. Not because software is just too intrinsically hard. Nope.

They fail because this stuff is just too damned interesting with too many ways to do things and too many things to do.

Andy blogs regularly at The e-Astronomer. Here are a few other examples of his work: