On the 26th of July in 1985, Donna Gartner arrived at the Ontario Cancer Foundation clinic for her 24th cancer treatment for carcinoma of the cervix.

The visit was in no way different from her prior 23, except that the clinic’s radiation therapy machine shut down 5 seconds into the treatment. The error shown on the screen indicated that no dose had been administered. The operator took the machine at its word and hit the “P” key to proceed.

The machine continued to error, and the operator retried the operation a total of 4 times, at which point a service technician was called, but they found no issue with the machine.

I’m sure some of you are already familiar with the radiation therapy machine in question, but just to be clear, Donna Gartner died in November of that same year. She was one of three deaths associated with the Therac-25.

The Therac-25 is a textbook example of dangerous software which was only tested in ideal conditions. The software was riddled with race conditions and lacked hardware interlocks that had been present in the Therac-20, so people who used this machine multiple times per day eventually became confident in their use, and fast as a result. Faster, unfortunately, than the software was designed to handle. A targeting plate that converted the stream of electrons produced by the machine into X-rays was not in place when the device was activated, resulting in massive overdoses of beta radiation.

Everyone living in a developed country now has at least some part of their life interfaced with software. This software doesn’t need to be obvious, like a mobile phone or a computer, it can be subtle, like how your information is handled by various departments of government.

Software has a strange implicit trust given to it, in spite of never earning such trust. Software is not a disciplined engineering field, and the vast majority of people writing software have no formal training in architecting software, and frequently no training at all.

We’ve given everything to software, and we only hold it to a standard once it breaks.

A prominent example of dangerously untested software is the Nest smart thermostat failure that left many users unable to control their thermostats during an exceptionally cold weekend in North America. I suppose you could call it lucky that it happened during a cold snap instead of a heat wave, and that older people are less likely to adopt this kind of technology since they’d be most at risk from extreme temperature fluctuations.

Another example can be found in the Nissan airbag software failure, which was mistakenly classifying some adults as children and not firing the air bags.

With the invention of smart meters, electricity companies can now completely automate shutting off of someone’s power if their bill is overdue. This is a horrifying development in the world of barely tested software. Any house fitted with a smart meter can have their electricity disconnected due to a software bug or slow processing in the company’s internal systems. For most of us, this is a mild annoyance that can be solved by a quick phone call.

For people with mains powered medical devices, it’s lethal. Humans need to follow due process when disconnecting power, including checking whether the house that’s being disconnected has any registered medical devices. Software can have those same checks built in, but in the world of barely tested software, there’s no regulations stating that such things need to be tested and proven.

So many things can go wrong in the chain: your ISP might be trying to intercept your pinned HTTPS calls and, when it fails, prevent you from paying via your banking application. The banking application you’ve used to pay the power company can glitch and say it was paid when it in fact wasn’t. The power company’s batch processing application that reads the list of payments that were made today might have recently been updated with buggy code that chokes on your last name because it has a diacritic, so it just silently errors and skips marking your account as paid up.

Programming desperately needs to become a mature field. Academic research into new programming languages is decades ahead of what is being used by developers in commercial environments. Universities no longer teach computer science to undergraduates, they teach programming because that’s what the commercial workforce wants them to teach. Undergraduate degrees in computer science are more often than not just certifications in the current most popular commercial programming language.

Most programming languages are unsound and unsuitable for writing life critical software, yet the languages of choice for most software was first released in the 70s and 80s. They can throw exceptions that go unchecked, attempt to access memory that is invalid and just completely override their own type system when asked nicely.

One alternative that was used before complexly typed languages had begun to mature was to rigorously test and retest the code, while using small subsets of the language that allow for less unsound behaviour. This was the approach taken by NASA. The problem with the approach is that even NASA was still writing buggy code, and at the time there was simply no better way.

Starting with Coq in 1989, programming languages with dependent types started coming into existence. I’m not going to try and explain why this is a good thing in full, but simply put, you can write proofs of the correctness of your code within the language itself. These features have taken so long to filter out of academia that it’s still considered the bleeding edge of software to use formal verification in critical applications.

This correct-by-construction approach is too mathematics heavy for the majority of programmers, I’ll admit. That said, when was the last time you met a building architect that built skyscrapers by stacking prefab houses 300 metres high, covering them with duct tape and praying they don’t fall over? I sincerely hope the answer is never, and that the analogy is clear.

This is a technical solution to a social problem. Governments do not regulate the architecture of software like they do with the architecture of buildings. This freedom allows capitalism to do what it does best: cut any corner it can. There’s no financial incentive to writing perfect software when mediocre software can be made faster and cheaper. Companies face negligble financial backlash when their software does fail, in comparison to the financial gain of making it to market first. Formal verification has taken so long to leave academia because safety doesn’t equal money.

We need governments to acknowledge the impact that software is already having on the world and impose critical software infrastructure legislation requiring that any software that can have devastating effects on human lives is formally verified, only then will articles like this be anything more than a talking piece.

Software is going to keep being a huge part of our lives, and I’m deeply terrified.

We need to stop letting carpenters design skyscrapers.