Beware the Bolognese

What can happen when spaghetti code is relied upon to operate mission-critical systems? We look at the Toyota spaghetti code debacle and how it could be the bad habits of programmers inheriting code at the heart of the problem.

The term ‘spaghetti code’ sends shared shivers down the spine of almost every developer. Tangled and complex in nature, it’s often described as unmaintainable, unstructured, impossible to debug… basically f**ked up.

As Michael O. Church explains, spaghetti code is an especially virulent but specific kind of bad code, and not to be confused or interchanged with meanings of bad code in general. For Church, its particular “badness” is instructive in how software is developed, because individual people rarely write spaghetti code on their own. “Certain styles of development process make it increasingly common as time passes”.

Instances of spaghetti code nightmares are not hard to come by online, with many a developer openly disclosing their experiences with a company’s screwed up source code. One of the most read about cases involves car manufacturer Toyota, whose egregious deviations from standard practice were exposed at trial.

In 2013, Toyota was deemed to have acted with “reckless disregard” in a lawsuit decision that involved the unintended acceleration (UA) of one of their vehicles. The case stemmed from the death of Barbara Schwarz and the serious injury of Jean Bookout, who lost throttle control of her 2005 Camry on an Oklahoma interstate highway.

SEE ALSO: NASA’s ten coding commandments

During the trial, software experts Philip Koopman and Michael Barr testified after looking into and dealing with Toyota’s software development process and source code, calling it “spaghetti-like”. Koopman, a safety critical embedded systems specialist, authored what is admittedly the textbook on embedded system design, and as a professor of computer engineering at Carnegie Mellon University completed more than 125 design reviews of industry embedded systems.

Barr, an embedded software specialist, spent almost two years reviewing Toyota’s source code under heavy security conditions, producing an 800-page report that was unapologetically damning towards Toyota’s engineering safety process. His and Koopman’s testimony proved to be the tipping point in determining the state of Toyota’s computer engineering practices.

As Michael Dunn reported, the overall critique of Toyota came down to the following points:

Toyota’s electronic throttle control system (ETCS) source code is of unreasonable quality

Toyota’s source code is defective and contains bugs, including bugs that can cause UA

Code-quality metrics predict presence of additional bugs

Toyota’s failsafes are defective and inadequate (referring to them as a “house of cards” safety architecture )

Misbehaviours of Toyota’s ETCS are a cause of UA

Toyota were also deemed to blatantly break their own coding standards, which they substituted themselves with little overlap of the industry standard. In a Safety Research & Strategies report, the testimony of Koopman and Barr documented important design process errors practically baked into the Toyota software model, including an absence of a peer code review.

Church’s definition of spaghetti code touches on this exact issue, among other flaws in the practice:

Spaghetti code is evil because (a) it’s a very common subcase of bad code, (b) it’s almost impossible to fix without causing changes in functionality, which will be treated as breakage if people depend on the old behaviour and (c) it seems, for reasons I’ll get to later, not to be preventable through typical review processes.

These reasons stem from spaghetti code forming incrementally, making it hard to catch in code review since each change that leads to “spaghettification” appears, at first, to be a net positive, according to Church.

While incremental code review is great at catching obvious bad coding practices, it doesn’t work for the spaghetti code strain. What is needed is a whole-program review, however there would be few businesses willing to conduct such a large-scale review operation. Funnily enough, Church comments here that this kind of practice is next to impossible over a long term period, “except among teams writing life- or mission-critical software, to ensure this high level of discipline in perpetuity”.

As a single developer dealing with spaghetti code, conducting such a review would be a daunting and almost futile process. However, quite often the additions to legacy code which eventually equal spaghetti code is written by us not them, according to Arialdo Martini:

There isn’t a single person who is responsible for the whole mess. The bad programmer, the Legacy Code Writer, is a fictional character we invented to justify our little, continuous, sins. I came to the conclusion that writing Legacy Code is a distributed activity. Each of us contribute, eventually not intentionally, to this crowded, incessant, slow process. Don’t waste your time trying to seek him: we are that programmer.

The problem is much bigger than just the programmers themselves. Processes need to be put in place to prevent spaghetti code from forming, lest every developer falls into the trap of blaming others for being forced to underperform. “But it’s not your fault. You are justified, since it’s the Legacy Code Writer’s fault”.

The problem with particles of Legacy Code is that they are like gems of corals: they tend to attract other bad code, and they soon become as hard as hell to remove. Introduce a temporary variable, and be sure that the next programmer will feel justified to attach some code to it.

Spaghetti code, as summarised by Church, is “a byproduct of industrial programming that is usually, but not always, an entropic result of too many hands passing over code”. Other than the issue of having unmaintainable code and an arbitrary program flow (plus having dire consequences for life-critical software), spaghetti code is also dangerous in the habits that developers can establish via their interaction with it.

How would we overcome the accumulation of these bad habits? Does it need to come from management, or are there ways for programmers themselves to undo the damage? Let us know in the comments below.