Lion Air crash: Jakarta Boeing 737 'had prior instrument error'

[floobydust]

To be fair, it's not like Rockwell Collins doesn't have a solid reputation and history developing avionics for airliners. It actually seems to me rather reasonable that an aircraft manufacturer would buy from a company that specializes in avionics. Now, folks are right that a system like MCAS, designed to protect the airplane, is probably harder/more risky to farm out than, say, nav equipment, which has pretty much nothing to do with the airframe.

Does Airbus not farm out work to subcontractors?

I'm baffled at how such a shit engineering job could be done with MCAS.
Rockwell Collins has produced quite the turd, despite their "reputation" as having avionics experience. There are multiple fatal errors in the S/W. I see no evidence of expertise writing the S/W or testing or certifying it.

Honeywell did a lot on the Airbus 320.

I guess Boeing and Airbus are just doing the airframe design and systems integration.

[KL27x]

I still don't know of any software bugs being admitted. The software might be doing exactly what Boeing specified, regarding the resetting after 5 seconds. Furthermore, it was after Boeing's own test flights that the response of MCAS was ultimately increased from 0.4 degrees to 2.5 degrees.

So it might have been a gradual creep into territory that eventually, in case of AOA failure, exceeded the plane's mechanical ability to move the stabilizer under that abnormal condition/load. Meaning AOA/MCAS failure can lead to a conditional failure/freezing of the stabilizer in a down position with little recourse unless you have several thousand feet of altitude you can afford to lose.

[SkyMaster]

To be fair, it's not like Rockwell Collins doesn't have a solid reputation and history developing avionics for airliners. It actually seems to me rather reasonable that an aircraft manufacturer would buy from a company that specializes in avionics. Now, folks are right that a system like MCAS, designed to protect the airplane, is probably harder/more risky to farm out than, say, nav equipment, which has pretty much nothing to do with the airframe.

Does Airbus not farm out work to subcontractors?

Of course they all work with subcontractors and partners. Boeing, Airbus, Embraer, Bombardier Aerospace; none of them is writting a single line of code. None of them is designed anything electronic either.

The large aircraft manufacturers do not even design the complete airframe themselves.

 

[djacobow]

Rockwell Collins has produced quite the turd, despite their "reputation" as having avionics experience. There are multiple fatal errors in the S/W. I see no evidence of expertise writing the S/W or testing or certifying it.

You don't know any of this. For all you know, Collins wrote software that performs flawlessly with respect to the specification they were given. Maybe the engineering was fine and the spec was bad. We just don't know.

[floobydust]

Na, Airbus and Boeing hire software engineers, look at their careers websites. Rockwell Collins seems to prefer new grads...

Is this is the mega corporation of the future?
Engineer-less, they just subcontract it all out.
Keeping with the MBA paradigm, get your nuts and bolts from the cheapest country and de-staff after the project is finished.

When the airplane(s) crash, the subcontractor's defense is "it wasn't in the requirements, nothing said to look at the second AoA sensor or let the pilot override it".

It's starting to smell like Volkswagen and Bosch, where Bosch made the ECU and software. Bosch wrote (requested) emissions defeat software for VW with a letter saying "for test purposes only" to absolve them. Bosch ended up paying $328M in that scandal.

Sure looks like MCAS went under the radar as something not critical, in order to avoid pilot training penalties, FAA scrutiny and going through basic safety tests.

[Nusa]

Na, Airbus and Boeing hire software engineers, look at their careers websites. Rockwell Collins seems to prefer new grads...

You do realize Software Engineer in most of the world, including the US, is just a title. It doesn't mean much by itself, and some people with that title don't have a degree of any kind. Your work history, experience, and qualifications are what matter in the end.

I retired with the title of Senior Software Engineer. It is backed by a Computer Science degree in my case, but many of my colleagues had degrees in other fields or never finished their formal education.

[SkyMaster]

Na, Airbus and Boeing hire software engineers, look at their careers websites. Rockwell Collins seems to prefer new grads...

They do no hire any software engineers to write code used in the commercial aircraft division. The software is bought, along with the hardware, from the various system suppliers.

[floobydust]

That's an issue - BSc in Comp Sci you are not bound by a code of ethics.

Engineering associations have been trying to include software in computer engineering, as something with formal documentation, calculations, that are subject to peer review. Just like a drawing that gets stamped, the PE stamps the software docs. I'm not sure it's realistic but something is needed to ensure critical software does not slither out the door.
The problem is you can never prove software correctness, it's impossible due to all the permutations of execution paths. IEC 61508 demands you've thought of the important ones, that kill people and that you have coverage and tested the modules and documented it all. I wonder if MCAS was even written to any S/W standard.

After these two airplane crashes, here it is the engineering profession that looks badly, as if they made stupid errors.
I suspect the executives and managers pushed too hard, circumventing regulatory processes etc. to hurry up the schedule and win the game.

[floobydust]

Na, Airbus and Boeing hire software engineers, look at their careers websites. Rockwell Collins seems to prefer new grads...

They do no hire any software engineers to write code used in the commercial aircraft division. The software is bought, along with the hardware, from the various system suppliers.

I get what you're saying, that the S/W is a commodity item in avionics. Just like ordering a nut, bolt, rivet.
But S/W is still unique to the particular aircraft, somewhere, someone has to know the system and tailor it. And take responsibility.
It's the S/W requirements, testing and certification - all three failed. Some scumbag exec gets MCAS classified as "non-critical".

I can understand Boeing and Rockwell Collins engineers keeping quiet and not blowing the whistle. Finding another job in town would be very difficult and their employment agreements are surely overreaching.

[FrankBuss]

The problem is you can never prove software correctness, it's impossible due to all the permutations of execution paths. IEC 61508 demands you've thought of the important ones, that kill people and that you have coverage and tested the modules and documented it all. I wonder if MCAS was even written to any S/W standard.

You can prove that a software is correct according to a specification, e.g. with systems like SPARK Ada. It is a lot of work, but it doesn't prove that the specification is right.

[chris_leyson]

You can prove that a software is correct according to a specification, e.g. with systems like SPARK Ada. It is a lot of work, but it doesn't prove that the specification is right.

Exactly. Maybe the MCAS software specification was never designed to deal with erroneus flight data.

[SkyMaster]

You can prove that a software is correct according to a specification, e.g. with systems like SPARK Ada. It is a lot of work, but it doesn't prove that the specification is right.

Exactly. Maybe the MCAS software specification was never designed to deal with erroneus flight data.

While it is possible to prove that a software meets the specification requirements. It is impossible to test/validate all the possible combinations of variables. Anything that is not specified in the requirements will react the way the programmer fantasized how he thought it should be. Also, every new software load always carry its own new set "undocumented changes".

This is true even with safety related software. And this is also true with flight control software. I am a speaking from personal experience.

I am not speculating that this is what happened with the MCAS. It seems that MCAS was piggy backed on top of the stick shaker system; where a single AOA sensor input is required for activation. The Boeing 737 was initially certified in 1967; it seems that anything could be certified 50 years ago. 50 years later, the Boeing 737 MAX is still riding on the initial 737 type certificate. The Boeing 737 type certificate data sheet is now at revision 61.

 

[Towger]

You can mathematically prove software with the likes of Z Notation https://en.m.wikipedia.org/wiki/Z_notation

Problem is it takes a lot work/time and to do and needs to done for the full stack.  There may be tools to help now, but 20+ yeas ago a simple loop resulted in pages of hand written equations.  I hated learning it.

Or course it does not stop a badly written specification.

[GeorgeOfTheJungle]

[KL27x]

^It started out as "they shoulda cut the stab trim." Now it's "they cut the stab trim, but they shoulda cut throttle." Next we'll find out "They shoulda cut stab trim, reduced throttle, and put a partridge in a pear tree." It's obvious that planes are only safe if this guy is your pilot.

AP cuts out. Sensor failures.

Pilot is trained to set pitch and throttle while figuring out the malfunctions. But pilot can't get the pitch up.

Speed increases, due to low AOA/pitch.

Cutting throttle would induce nose down. Need the throttle to maintain nose up, plus higher airspeed gives more lift at this low pitch.
But higher air speed plus the nose up force from the engines locks up the stabilizer.

At high enough airspeed with column pulled back, MCAS can easily move the trim down. But the pilot can't move the trim back up unless he does the "roller coaster maneuver." But the window for that maneuver is already gone. Plane going too fast at this point, and not enough altitude.

Sounds like a Chinese fingertrap. Or a Sandra Bullock movie with a bus.

In both crashes, the plane was going abnormally fast. IF this hypothetical can occur, and if the plane were to get into this spot, I wonder if extending the flaps would be a good idea. To increase the lift and drag* to counteract the increase in speed, while increasing lift to preserve/gain precious altitude, and without having to cut the engines as much (and getting the nose down force that would ensue). The flaps are retracted to increase fuel efficiency at cruise. But maybe there's an obvious reason that this is a bad idea, like the flaps would break off if you extended them at high speed.

*This is essentially what a high AOA does for the plane... to increase lift at the cost of increased drag. If you can't get the nose up, get the flaps out?

[HighVoltage]

It seems like the Airbus A330-300 had a similar problem already many years ago, when false info were sent to the main flight control computers.

 

[KL27x]

^That was an actual software bug. The sensor malfunction created a situation that was undiscovered/untested and in this situation, the data got rolled over, or something like that, instead of clipped. The problem wasn't the sensor failure, it was an honest to goodness software bug. Notice that on this plane the FCC has control over the elevator, not just the rudder. The FCC can bounce people off the ceiling. In the 737, the FCC/MCAS can't do that.

With the MAX, the primary problem could be the aerodynamics of the plane and thrust axis of the engines combined with poor compromises in the implementation of the corrective automated response. The pilots of the MAX remained calm and did not panic, just as they were trained, calmly working through a checklist. Flying a big passenger jet, this should always be the correct attitude. But this situation could be more dire than Boeing has let on, and it might call for immediate and more drastic action before the problem spirals beyond a point of no return. And appropriate training.

I'm not suggesting one mistake is worse than the other. If the Airbus crashed, a software bug would have killed all the passengers, and they would have paid. But then they could fix the bug and carry on. The MAX already has bigger issues from a plane manufacturer/airline perspective.

[tooki]

ALL 737 planes have the flight characteristic that MCAS is designed to address. It’s merely stronger in the MAX than in earlier versions, so MCAS is there to compensate so that pilots don’t need to recertify. It’s incorrect to think of it as an unstable aerodynamic design, it’s just got a character that’s different enough that they couldn’t plausibly claim it under the same airworthiness certificate.

[SkyMaster]

ALL 737 planes have the flight characteristic that MCAS is designed to address. It’s merely stronger in the MAX than in earlier versions, so MCAS is there to compensate so that pilots don’t need to recertify. It’s incorrect to think of it as an unstable aerodynamic design, it’s just got a character that’s different enough that they couldn’t plausibly claim it under the same airworthiness certificate.

You wrote "so MCAS is there to compensate so that pilots don’t need to recertify.", but I think you meant "...so that pilots don’t need to retrim".

 

[Brumby]

No - I think he is referring to the avoidance of having to acquire a new Type Rating.

[Nusa]

The statements are not mutually exclusive. Both can be true. And apparently it works well when nothing is broken. The problems are all in the area of inadequate redundancy, fault-handling and fallback behavior.

[tooki]

No - I think he is referring to the avoidance of having to acquire a new Type Rating.

Exactly this. Without MCAS, the 737 MAX would have different stabilizer trim needs, thus requiring a new type rating for the aircraft and thus requiring pilots to acquire it.

The statements are not mutually exclusive. Both can be true. And apparently it works well when nothing is broken. The problems are all in the area of inadequate redundancy, fault-handling and fallback behavior.

Exactly.

[KL27x]

The emergency procedure may not be adequate.

If you look at Air Alaska 261, the plane was put in a similar situation. It started out with a seized jackscrew nut. The place where the trim seized, the pilots had to pull up on the yoke with ~10 lbs of force to keep the plane level.

After futzing with the trim controls on/off for 15 minutes, the nut finally unseized. But because it was stripped and because they were pulling up on the yoke, the stabilizer moved down, rapidly, where it jammed again.

The pilots eventually regained level flight by pulling on the yoke with 140 lbs of force. They were able to communicate with ATC and were preparing to attempt a landing at LAX.

But the jackscrew was not designed to handle this "unusual position" of the stabilizer and the entire jackscrew assembly broke from the stress. The pilots elevator controls were now useless. MCAS malfunction puts the stabilizer into an "unusual position." Not being able to physically turn the trim wheel, manually, seems like the pilots need to workout. But this is enough force to actually break the entire mechanism (on a different plane, of course).

Roller coaster maneuver is now known to many more pilots, and hopefully there is room to do it, if needed.

[floobydust]

Airbus' use of three AoA sensors has had incidents and crashes.
Two of the three AoA sensors agree but are both faulty, example is a pair seizing due to ice, and then stall protection activates.

XL Airways Germany Flight 888T did not make it. BEA report with AoA tests showing sticking from moisture in the bearings freezing.

Lufthansa Flight LH1829, the fix was to break AoA1 and AoA2 consensus by shutting off Air Data Reference unit (ADR) 2.
AoA3 was reading correctly, so all that was left is a disagreement.

There's a real need to over-ride anti-stall automation, and not just switch it off leaving you in a dive.
I wonder what Boeing's new MCAS software does, given it is full of unicorns and rainbows and the 96 test flights so far, and many billions of dollars and 346 lives lost.
We all know end-use testing of software is not proof of correctness, 1,000 flights would still say nothing really. How does it do with a blocked pitot tube or airspeed sensor problems? All inputs to the S/W module have to be considered going faulty.
I have zero faith in Boeing or Collins being able to do a proper fix. Good luck selling it to the globe.

[KL27x]

It's interesting that in the Lufthansa flight, individual AOA sensor data can be switched off. So if the pilot discovers two sensors out of three are broken, he can potentially switch off the ones that are faulty based on what he is perceiving. This was done during a call to the maintenance department for the airliner, so it's not necessarily a "memory item," but it is at least possible. Maybe Boeing will be able to add this ability to the 737. In ET302, both pilots called out "left alpha vane," so they were aware not only of an AOA disagree; they actually recognized which sensor was incorrect. 

In the first link, 888T, the faulty AOA sensors were caused by improper maintenance. And it appears the main reason the plane ultimately crashed is because the pilots were testing the stall warning, but they were not aware that the AOA sensors were malfunctioning. And unfortunately, they did this test at a low altitude after being denied the air space they wanted. Trusting safety technology a bit too much, maybe. The pilots obviously trusted the plane and were demonstrating its condition to the leasor. And maybe they had done this test, previously, in preparation for this demonstration flight. But they didn't take into consideration that it had just had a complete overhaul.