June 27, 2019

Boeing's Software Fix For The 737 MAX Problem Overwhelms The Plane's Computer

The Boeing 737 MAX continues to be a troublesome airplane.

Two crashes of the plane type, which cost the lives of 346 people, revealed a significant problem not only with the messed up Maneuvering Characteristics Augmentation System (MCAS).

It then turned out that the manual trim wheels which Boeing advised to use to counter MCAS are impossible to move when needed. Moon of Alabama detailed the problem back in May and last week the Wall Street Journal confirmed the issue. This also affects the older Boeing 737 NG.

While that problem has still not been solved a new one came up.

Boeing promised to release a software fix for the MCAS by April 2019. But that turned out to be more difficult than thought. Three month later there is still no final fix available. Meanwhile a new problem that will cause further delays was revealed only yesterday:

In a flight simulator last week, F.A.A. pilots tested erroneous activations of anti-stall software that pushes down the nose of the Max, two people with knowledge of the matter said. The software, known as MCAS, was involved in two crashes that killed 346 people. In at least one instance, an F.A.A. pilot was unable to quickly and easily follow Boeing’s emergency procedures to regain control of the plane. The pilot rated that failure as catastrophic, meaning it could lead to the loss of an aircraft midflight, the people said.

...

The issue discovered last week is linked to the data-processing speed of a specific flight control computer chip, according to the two people with knowledge of the matter. In the test, the F.A.A. pilot encountered delays in executing a crucial step required to stabilize an aircraft.

It seems that the additional signal processing and calculations needed for the MCAS fix overload the Flight Control Computer's (FCC) processor and delay its reaction.

Boeing has been developing a software update for the Max for eight months, [a Boeing spokesman] said. It is unclear whether the new flaw can be resolved by reprogramming the software or requires a hardware fix, which would be costlier and could take much longer.

The 737 MAX has, like the previous 737 NG and Classic versions, two FCC's which each have two Central Processing Units (CPUs).

737 Flight Control System



bigger

As the former Boeing flight control engineer Peter Lemme wrote last year in a technical note of the issue:

Each FCC is comprised of two processors, each of which perform independently. Each FCC has two 16-bit CPUs. The two processors have different part numbers to make sure that a design problem is not in both processors. The CPUs calculate different commands. ...

In another note Lemme wrote:

The 737 FCC installation is a "dual-dual" configuration. Within each of the two autopilot computers there are two different processors, that each themselves are programmed by different people. The greatest threat is a common-mode software failure. Having two different groups program from a common set of requirements is a means to diminish a common mode failure.

...

The 737 dual-dual architecture is very unique. The decision to make speed trim single channel, single processor goes back to the 737 classic. The MCAS function is just another FCC software module that behaves, at a high level, like speed trim, whose architecture would have then been replicated.

The 737 uses only one FCC at a time and the Speed Trim System (STS), of which MCAS is a part, runs only on one of that Flight Computer's two internal processors.

The processors in question are said to be Intel 80286 type CPUs. The original Intel version of that CPU, sold between 1982 and 1991, had a maximum clockrate of 4, 6 or 8 MHz. It was later manufactured by a number of other firms, including by AMD and aeronautics company Harris, with a clockrate of 20 and 25 MHz. It is likely that the Boeing 737 FCC uses these or similar types.

These old processors are very reliable and error free. But they have less than 1/1000nds of the computing capacity of a modern cell phone. According to Lemme one CPU in the flight computer runs up to 11 different processes. All need to receive the input of external sensors, run through their algorithms, and signal a command to the relevant actuators that control the moveable flight surfaces of the plane. That the FAA pilot "encountered delays in executing a crucial step" caused by the computer points to a capacity overload.

Some decades ago your host programmed special input device drivers for Intel 80286 and alike systems. Their purpose was to record and process data from industrial process sensors, often hundreds at a time. Performance and timing issues required that the 80286 input drivers had be written in low level assembler language. But even with extremely optimized code the system would eventually come to its limits. The delayed procession of data from one sensor would eventually cascade into further delays and in the end the system would fail to record and process anything. The task was simply above its physical limits.

Flight control computer run special operation systems with minimal overhead. They are programmed in highly efficient programming languages. The software design and implementation follows a very strict process using specialized tools (see Green Hills' products for examples). All these are much better than what I used during my programming times.

Programs written for flight control purposes are already highly optimized. To further optimize them 'by hand' would break the regulated process that production of such software requires.

Boeing says that it can again fix the software to avoid the problem the FAA just found. It is doubtful that this will be possible. The software load is already right at the border, if not above the physical capabilities of the current flight control computers. The optimization potential of the software is likely minimal.

MCAS was a band aid. Due to the new engine position the 737 MAX version had changed its behavior compared to the older 737 types even though it still used the older types' certification. MCAS was supposed to correct that. The software fix for MCAS is another band aid on top of it. The fix for the software fix that Boeing now promises to solve the problem the FAA pilot found, is the third band aid over the same wound. It is doubtful that it will stop the bleeding.

The flight control computers the 737 MAX and NG use were developed in the early to mid 1990s. There are no off-the-shelf solutions for higher performance.

Boeing's latest announced time frame for bringing the grounded 737 MAX planes back into the air is "mid December". In view of this new problem one is inclined to ask "which year?"

---

Previous Moon of Alabama posts on Boeing 737 issues:

Posted by b on June 27, 2019 at 18:41 UTC | Permalink

Comments

next page »