Boeing 737 MAX and Software's Uncanny Valley

On March 10th, the 2nd Boeing 737 MAX crashed in just under 6 months. The catastrophe has caused fear, speculation of the cause, and the grounding of the entire 737 MAX fleet of planes in all countries that fly them.

This is a tragic event that has caused many deaths, and we still don’t know the exact cause. If more details come out, it might completely change the narrative but for now there’s one particular angle that I’m interested in: the computer/software + human component of this story.

Boeing’s New Aircraft

This new aircraft model, the Boeing 737 MAX 8, is the successor to the previous 737 which first flew in 1967. One of the changes is the addition of the system that is now being focused on as having caused the first crash and possibly the latest. The system is called MCAS, or if you enjoy enterprise programming: Maneuver Characteristics Augmentation System.

This new system was added due to a difference in how the plane was designed which changed how it handled in air. The system, when enabled, would hide this new flight behavior behind a layer of software abstraction that emulated the way the previous 737 flew. This was effectively using upfront software costs instead of having to retrain pilots to handle this new behavior.

A few weeks after the first crash in November of last year, the Wall Street Journal reported:

Boeing marketed the MAX 8 partly by telling customers it wouldn’t need pilots to undergo additional simulator training beyond that already required for older versions, according to industry and government officials. One high-ranking Boeing official said the company had decided against disclosing more details to cockpit crews due to concerns about inundating average pilots with too much information—and significantly more technical data—than they needed or could digest.

It isn’t hard to see the problem with this: in the event the system malfunctions, the pilot still needs to know that the MCAS is malfunctioning and how to turn it off.

This is very common working at a tech company. You have automated scripts or systems that help gather diagnostic information, logs, or any other information during an outage to debug what is happening. This works great — until your tools don’t work.

When this happens, you need to diagnose the problem manually and if you don’t know where or how to find that information, you are left in pretty bad shape and with a system that is broken.

This appears to be what happened with the first crash back in October 2018. It was determined after the event that the aircraft crashed due to a malfunctioning sensor that sent incorrect data to the MCAS system. The system was trying to respond to this incorrect data but the pilots didn’t know what the aircraft was doing, so they tried to counter its response and never turned off the system which resulted in the crash.

An excellent article in Air Facts by Mac McClellan called Can Boeing Trust Pilots? looks at the latest news with way more detail and knowledge that McClellan, an experienced pilot, can provide. It is a very intriguing read.

These few things stood out to me from McClellan’s article:

What Boeing is doing is using the age-old concept of using the human pilots as a critical element of the system. Before fly-by-wire (FBW) came along, nearly all critical systems in all sizes of airplanes counted on the pilot to be a crucial part of the system operation.

Just for some background without getting too much into the weeds: the term fly-by-wire (FBW) is similar to analog vs digital.

Before fly-by-wire, everything was mechanical but with it interfaces are converted to flight controls digitally. This means a computer can manipulate controls and fly the plane without a human intervening to move levers, wheels, peddles, etc.

And then a bit later:

FBW removes the pilot as a critical part of the system and relies on multiple computers to handle failures. Boeing is now faced with the difficult task of explaining to the media why pilots must know how to intervene after a system failure. And also to explain that airplanes have been built and certified this way for many decades. Pilots have been the last line of defense when things go wrong. What makes that such a tall order is that FBW airplanes – which include all the recent Airbus fleet, and the 777 and 787 from Boeing – don’t rely on the pilots to handle flight control system failures. FBW uses at least a triple redundant computer control system to interpret the inputs of the cockpit controls by pilots into movement of the airplane flight controls, including the trim. If part of the FBW system fails, the computer identifies the faulty elements and flies on without the human pilots needing to know how to disable the failed system.

So what this comes down to is that Boeing already can handle these failures on other more advanced models of aircrafts. Since the past version of this system required pilot intervention in the original 737, the current 737 MAX also required the pilot to intervene.

This left us in this state:

The aircraft was improved with this new system, the MCAS, to retain flight characteristics of the original 737

This new system, which meant to be a minor version bump, actually required additional training of the pilots. They introduced a backwards incompatible change and broke the pilot’s API

They made these choices to make the transition to the new aircraft easier for the pilots, who are, by the way, professionals trained extensively and have years of expertise.

This is where the issue of human and computer interaction starts to get fascinating.

Human & Computer Assistance

There was an interesting point that Sam Altman made on the Conversations with Tyler podcast:

You know, when AI started beating humans in chess, there was a short period of time where the very best thing of all was a human and an AI playing chess together. The AI would say, “Here’s six moves.” The human would pick the best one of those, and that was better. That could beat an unassisted… that merged version—it’s not really a merge—that teamed-up version could beat an unassisted AI. I don’t know exactly how long that stayed true for, and people loved that fact, but it didn’t stay true for long. The humans started making the AI worse than the AI was playing alone as it got smarter. I think we will learn that we’re just not that smart. The size of a human brain has all of these biological limitations, but you can make a really big computer with very fast interconnects between the chips.

This brings us to the idea that as a thing improves, there may be pitfalls, or valleys, where the improvement can actually get worse depending on the human’s perception or interaction with it.

This is well known in the field of robots and 3D animation where it is called the “uncanny valley.” It hypothesizes that as the appearance of an object gets closer and closer to a human, it goes through a valley where humans become repulsed or disturbed by its appearance.

This repulsion continues until the appearance improves enough and the unpleasant perception gives way to a very human-like appearance.

At the core of uncanny valley is this: human perception. This isn’t too far from the reality we are in now with computer assistance. Given humans can misperceive the physical resemblance of an object to a human, it could also follow that a human will misperceive the intelligence of a computer or system relative to that of their own intelligence.

This can happen in two ways:

A human might design the computer to ask the human for input and the computer might continue to ask even if its capability exceeds the human’s. In this case, the computer is designed to babysit the human by the developers.

A human might override the decisions that a computer might make, even if the computer outperforms the human. In this case, the human thinks it is babysitting the computer.

In both of these cases, when a computer is designed to ask for human input, or a human assumes it knows more than the computer, it can perform worse on a task compared to if the computer just operated on its own volition.

It’s this “nannying” that humans feel necessarily to do with complex computer software, which just ends up putting the outcome in a suboptimal, local minimum.

This becomes the “nanny valley” and looks like the following:

I suspect that this valley occurs on the curve at different locations depending on the task at hand (chess, flying, driving) as well as the skill of the human involved.

If Magnus Carlsen used a computer to assist him with chess, it is a lot different than if I were to use a computer. I know the rules and can think a few moves ahead but don’t know any strategy or tactics. I would probably have the same performance if I were to instead roll a die and randomly pick one of the computer’s options.

In the case of the latest 737 MAX crash, we still don’t know exactly what happened but it’s clear in the past that trade-offs were made in the design of the MCAS to change the way the aircraft handled and it was abstracted away behind software that would need to be disengaged if issues arose.

As we develop more advanced software that starts to automate away tasks that humans used to do such as driving planes, driving cars, factory work, and many other tasks, this will always be a problem we will face.

This tragedy illustrates an important point about the interaction between humans and computers: a human’s perception of their own skill relative to a computer’s can sometimes cause us to make worse decisions. This choice can leave us all in this “nanny valley” and as a result, we are all less safe.