We occasionally have students ask for help on software, “My software is perfect, but it doesn’t work!” Your software is never perfect. My own software is never perfect.

I recently found that I made someone’s top ten list with regard to software that I had written 37 years ago. It’s not a top ten list anyone would aspire to be listed on. Software that I wrote in 1979 is number two on this list of ten historical software bugs with extreme consequences. I learned some very important lessons from that experience.

Background

My first job out of college was to document what everyone thought was a very complete and very well tested set of software for processing low level data solar backscatter ultraviolet / total ozone mapping system (SBUV/TOMS) on the NASA’s Nimbus 7 satellite. The entire development had already moved on to other projects with different employers. They left behind a large set of code and a very tall stack of computer printouts that contained their test results.

I started from the state of “What is this ‘FORTRAN’ language?” but quickly proceeded to “How can this code possibly work?” and from there to “There’s no way this code can possibly work!” I finally looked at that massive stack of test results on reaching that final stage of understanding. I was the first to do so except for the developers who had abandoned ship. Nobody else had looked at those tests results. They instead looked at the amazing thickness of the printouts.

Testing by thickness always has been and always will be a phenomenally bad idea. Some of those test printouts were slim; these were failed compilations. The rest were what were then called “ABEND dumps.” In those days, the equivalent of what is now called a segmentation fault resulted in the entire virtual memory for the process in question being printed out in hexadecimal. The result was a huge waste of paper. (The modern equivalent is a segfault and core dump.) Not one test indicated success.

This turned out to be a career maker for me. I made a name for myself by fixing that mess. As a result, I was subsequently given the privilege of working directly for the principal investigator of that pair of instruments and his team of scientists. Instead of the low-level computer science stuff involved with my first task, my next task truly did involve scientific programming.

Why the Nimbus 7 satellite did not discover the ozone hole

Of the two ozone measuring instruments on the Nimbus 7 satellite, one (SBUV) had been flown previously, but the more precise instrument (TOMS) was brand new. The previously flown instrument sometimes yielded flaky results when the solar angle was low, and the team scientists were worried that the same would apply to this newer instrument. The scientific team did not want their good scientific names sullied by suspect data. As a result, they vehemently insisted that I filter out those suspect results by resetting data where the solar incidence angle was low and where the estimated ozone quantity lay outside a predetermined range to a value that meant “missing or invalid data”.

I argued that if I did what he asked that there would be no way to discover anomalies. “We can change your code if we discover anomalies.” I suggested that I produce two products, an unfiltered one for NASA internal use only and a filtered version for release to the wider research community. They did not want any part of that, either, on the basis that the unfiltered version would somehow get outside of NASA. “Do what I told you to do, or we will tell your employer to assign someone else to us.” I capitulated and did what he told me to do.

Karma!

The Nimbus 7 satellite did not discover the ozone hole. Credit for that discovery instead goes to Joseph Farnam, who simply pointed a device invented in the 1920s (a Dobsonmeter) up into the sky. Mr. Farnam received a very nice obituary in the New York Times. The SBUV/TOMS team will more or less die anonymously. That’s karma.

As I should have been more insistent with the scientific team, I too was stricken with karma. In 1986, curious minds at NASA wanted very much to know why their very expensive satellite did not discover what a person using a 1920s era device had discovered. The scientific team discovered that my name was all over the code that hid the ozone hole from NASA. (They conveniently forget why this was the case.) This made people high up in NASA want to talk to me, personally. Despite having switched employers three times and despite having moved 1400+ miles away from that initial job, I received numerous phone calls and even a few visits from people very high up in NASA that year. I told them why that code existed, and also how to fix it. Voila! After reprocessing the archived satellite data, the Antarctic springtime ozone hole appeared every year.

What I learned