Lion Air crash: Jakarta Boeing 737 'had prior instrument error'

[floobydust]

It was said (original) MCAS toggled between using the left or right AoA sensor, every flight. That would be another clown algorithm that could have confused the maintenance crew as to which sensor was malfunctioning. Imagine charging $80,000 for the AoA DISAGREE annunciator option which didn't even work.

At least the AoA sensors are on the list somewhere with the European regulator:
"EASA’s checklist includes a number of issues that have been disclosed: the potential difficulty pilots have in turning the jet’s manual trim wheel, the unreliability of the Max’s angle of attack sensors, inadequate training procedures, and a software issue flagged just last week by the FAA pertaining to a lagging microprocessor. But the agency also listed a previously unreported concern: the autopilot failing to disengage in certain emergencies."

https://www.bloomberg.com/news/articles/2019-07-05/europe-sets-out-demands-for-boeing-before-max-can-fly-again

I have no idea what the new autopilot problem is about.

[m98]

MCAS toggled between both AoA sensors before each flight? Wow, those 9$ programmers really wanted to retaliate for their bad salary...

I can imagine how the MCAS software got bloated up so much that it would slow down the computers it runs on: model based design. Drag and hack together the control system in Matlab/Simulink and then use the auto-generated code. I don't know if Boeing does this on their civil airliners, but it is a trend in the industry.

[KL27x]

MCAS toggled between both AoA sensors before each flight? Wow, those 9$ programmers really wanted to retaliate for their bad salary...

This is pretty stupid, yeah. Doing it like this means that the more sensors you have, the more the chance of a malfunction. If MTF is 100,000 flights, then 1 sensor might be good for 10,000-1,000,000 flights. 2 sensors, and the MTF that 1 of them will fail will decrease.

You could have 100 sensors, and it means that if 99 of them are good, you're eventually gonna crash the plane.

This is essentially using a $100 million plane and 100+ souls as an AOA sensor tester.

I doubt this has anything to do with the $9/hr coders, though. They would surely not be in charge of designing the algorithm. You don't ask a coder to design an airplane anti-stall system, do you? You pay them to implement it to w/e specs your team of airplane engineers and test pilots and/or the FAA have deemed best/necessary, right? And $9.00 an hour is probably a pretty good wage in India. So it was 99.9% white guys that fucked this up.

[floobydust]

Found the source:
"Since MCAS is an FCC function, the AoA source for MCAS is that of the FCC in use; ie FCC 1 uses the Captains AoA probe and FCC 2 uses the F/Os AoA probe. When the 737 is powered up the FCC used is FCC 1 for that flight, this changes for each subsequent flight until the aircraft is powered down. Therefore the AOA sensor that is used for MCAS changes with each flight post power-up." http://www.b737.org.uk/mcas.htm#aoa
"... it [MCAS] starts on the left, then alternates between left and right on each flight." Reuters

So, after a cold-boot MCAS is going to use the left AoA sensor. Russian roulette has better odds.
In JT043 (and previous flight), JT610, ET302 it was the left AoA sensor that had problems. It's strange, almost as if the left resolver-to-digital has issues.

After JT043, the left (Captain's) AoA sensor had errors and apparently was replaced by a refurb, sensor came from a repair shop in Florida, XTRA Aerospace Inc.
Bloomberg seems to sensationalize the stories and not correct conflicts, earlier saying the sensor was not actually repaired/replaced, just a reboot to clear trouble codes.

[MyHeadHz]

Then the problem of the toyotas was that they had an accelerator pedal... ??

This is a bit off-topic, but people get that wrong a lot...

The problem with the Toyotas was that they had no kill switch or other emergency bypass.

With any other vehicle, there is always a way to kill and/or disengage the engine from the rest of the drive train- and almost always both.  Simply depress the clutch (or put it into neutral in an automatic or manual) or turn the physical key off- two emergency solutions.  Hell, even on cheap dirt bikes and kids' go-karts there are kill switches and/or dead-man switches.  In that 'keyless' model of Toyota, there was neither way to physically turn the engine off nor disengage the engine from the rest of the drive drain in the event of a runaway engine.  For some reason, the engineers decided to lock the engine and drive train while throttle was being depressed- making it impossible to physically force it out of gear.  There was no emergency cutoff to kill the engine at all, or any other way to kill the engine or stop the vehicle.  In competing models from other brands, there are ways to disengage the engine under load.  The engine might die, but you won't.

There was some marketing scapegoat BS they shoveled out about some stupid "floor mat", but that is irrelevant (if it was true at all).  If the vehicle had standard safety capabilities (such as those mentioned above), that family would still be alive.  There are a lot of causes for an engine to lock at full throttle, but there is no excuse to remove any way to safely deal with that problem when it comes up.  Toyota got cocky by removing those critical safety features and it backfired hard.  Then they deflected the blame to the innocent deceased police officer (the driver).  Toyota can suck a fat one.

/rant

[G7PSK]

Then the problem of the toyotas was that they had an accelerator pedal... ??

This is a bit off-topic, but people get that wrong a lot...

The problem with the Toyotas was that they had no kill switch or other emergency bypass.

With any other vehicle, there is always a way to kill and/or disengage the engine from the rest of the drive train- and almost always both.  Simply depress the clutch (or put it into neutral in an automatic or manual) or turn the physical key off- two emergency solutions.  Hell, even on cheap dirt bikes and kids' go-karts there are kill switches and/or dead-man switches.  In that 'keyless' model of Toyota, there was neither way to physically turn the engine off nor disengage the engine from the rest of the drive drain in the event of a runaway engine.  For some reason, the engineers decided to lock the engine and drive train while throttle was being depressed- making it impossible to physically force it out of gear.  There was no emergency cutoff to kill the engine at all, or any other way to kill the engine or stop the vehicle.  In competing models from other brands, there are ways to disengage the engine under load.  The engine might die, but you won't.

There was some marketing scapegoat BS they shoveled out about some stupid "floor mat", but that is irrelevant (if it was true at all).  If the vehicle had standard safety capabilities (such as those mentioned above), that family would still be alive.  There are a lot of causes for an engine to lock at full throttle, but there is no excuse to remove any way to safely deal with that problem when it comes up.  Toyota got cocky by removing those critical safety features and it backfired hard.  Then they deflected the blame to the innocent deceased police officer (the driver).  Toyota can suck a fat one.

/rant

Yes throttle sticking is nothing new and can happen to any engine/vehicle, I had that happen to me back in the mid 70's with a MK2 Jaguar the cable jammed with the throttle half open in the centre of Cambridge while in 2nd gear I was up to nearly 60MPH before I turned the ignition switch off. So kill switches are an imperative at all times.

[MT]

FAA let Boing sign their own safety certs! 

https://www.zerohedge.com/news/2019-07-27/faa-let-boeing-sign-its-own-safety-certifications-737-max
https://www.nytimes.com/2019/07/27/business/boeing-737-max-faa.html

[Marco]

Reverse Monty hall.

[SiliconWizard]

FAA let Boing sign their own safety certs! 

https://www.zerohedge.com/news/2019-07-27/faa-let-boeing-sign-its-own-safety-certifications-737-max
https://www.nytimes.com/2019/07/27/business/boeing-737-max-faa.html

A double-edged sword. That would also hand off all responsibility to Boeing.
I bet it's not going to end up very pretty.

[Bud]

FAA let Boing sign their own safety certs! 

https://www.zerohedge.com/news/2019-07-27/faa-let-boeing-sign-its-own-safety-certifications-737-max
https://www.nytimes.com/2019/07/27/business/boeing-737-max-faa.html

Hows that being the news? This was known from day 1. 

[MT]

FAA let Boing sign their own safety certs! 

https://www.zerohedge.com/news/2019-07-27/faa-let-boeing-sign-its-own-safety-certifications-737-max
https://www.nytimes.com/2019/07/27/business/boeing-737-max-faa.html

Hows that being the news? This was known from day 1. 

No idea , maybe MSM after all just ""fake news""?         ...

[raptor1956]

FAA let Boing sign their own safety certs! 

https://www.zerohedge.com/news/2019-07-27/faa-let-boeing-sign-its-own-safety-certifications-737-max
https://www.nytimes.com/2019/07/27/business/boeing-737-max-faa.html

De-regulation has been a major thing for 4 decades and this is the logical consequence of this mindset.  The FAA has been impeded in there regulatory efforts by a combination of this push to de-regulate and the controls on there purse strings that limit the number of people that might otherwise investigate and test things.  When you have too few people it makes sense to outsource this testing and regulatory work and who better to do that then the people designing and building the planes -- I mean, who know the system better than them.  So, the long-term consequence of this de-regulation mania is that we put the fox in the hen house with predictable results.


Brian

[CiscERsang]

But MCAS (the Maneuvering Characteristics Augmentation System) per se is not the problem, the problem is whatever it is that makes it (mal) function not-as-intended, be it the sensors, the circuitry associated with the sensors, or a bug à la toyota in the software.

partially agree..
As a side note: to keep nose within a certain angle tolerance, the system obeys the PID controller rules.

There's a complex issue there, I guess;
-unknown external factor(s) which is just swinging that system based on PID controller rules (here could be many sub-items)

• attack angle sensor is OK, however, in certain conditions its signal cannot be processed properly by system (air flow whirl at place where sensor(s) mounted, micro/macro swinging of fuselage or whatever)
• attack angle sensor is NOK, low reliability
• PID controller is not capable to process exceeding certain limits incoming values, due to wrong coefficient settings or so;
• it is not possible in principle to design an appropriate PID system, due to exceeding all limits external factors (e.g. much unbalanced fuselage)

-missing of elementary redundancy (that not the case, I think, it's aviation);
-influence redundand control channels on each other, somehow (code, conception mistake);
-a lack of option to override that system (code, conception mistake);
-no docs and corresponding training program on that system behaving and misbehaving;

BR

Generally speaking, the PID regulation rules it's foundation of automation based on 'higher mathematics' and constant head ache by students.

[BravoV]

FAA let Boing sign their own safety certs! 

https://www.zerohedge.com/news/2019-07-27/faa-let-boeing-sign-its-own-safety-certifications-737-max
https://www.nytimes.com/2019/07/27/business/boeing-737-max-faa.html

De-regulation has been a major thing for 4 decades and this is the logical consequence of this mindset.  The FAA has been impeded in there regulatory efforts by a combination of this push to de-regulate and the controls on there purse strings that limit the number of people that might otherwise investigate and test things.  When you have too few people it makes sense to outsource this testing and regulatory work and who better to do that then the people designing and building the planes -- I mean, who know the system better than them.  So, the long-term consequence of this de-regulation mania is that we put the fox in the hen house with predictable results.


Brian

Sorry, don't mean to derail, its just can't help it to think about "other" safety/regulatory bodies that are in charge for simpler daily stuffs, say like ... food safety ? tap water ? medicine ? construction ? etc ...

Well, may be its just me, I had goose bumps just to think about it.

[SilverSolder]

It is probably a balancing act between reasonable regulation and "getting in the way of progress".

Striking a good balance in anything is always a challenge...

[Kleinstein]

But MCAS (the Maneuvering Characteristics Augmentation System) per se is not the problem, the problem is whatever it is that makes it (mal) function not-as-intended, be it the sensors, the circuitry associated with the sensors, or a bug à la toyota in the software.

partially agree..
As a side note: to keep nose within a certain angle tolerance, the system obeys the PID controller rules.

There's a complex issue there, I guess;
-unknown external factor(s) which is just swinging that system based on PID controller rules (here could be many sub-items)

• attack angle sensor is OK, however, in certain conditions its signal cannot be processed properly by system (air flow whirl at place where sensor(s) mounted, micro/macro swinging of fuselage or whatever)
• attack angle sensor is NOK, low reliability
• PID controller is not capable to process exceeding certain limits incoming values, due to wrong coefficient settings or so;
• it is not possible in principle to design an appropriate PID system, due to exceeding all limits external factors (e.g. much unbalanced fuselage)

-missing of elementary redundancy (that not the case, I think, it's aviation);
-influence redundand control channels on each other, somehow (code, conception mistake);
-a lack of option to override that system (code, conception mistake);
-no docs and corresponding training program on that system behaving and misbehaving;

BR

Generally speaking, the PID regulation rules it's foundation of automation based on 'higher mathematics' and constant head ache by students.

From the descriptions given, the MACS system is much simpler than PID, more like a simple on/off if hitting the limit. Per se this may not be a problem  - though for me as a non pilot I don't see how this would make the new plane to be more similar to the old one and avoid extra training. To me it makes things worse not better, even if working as intended. The real problem are additional faults - some of them really blatant:
1) not using redundancy, though redundant hardware is there.
 There was the optional check program to indicate sensor discrepancy - but this did not work in most cases.
2) not checking for non plausible sensor inputs - especially important when using only 1 of the sensors.
3) not telling and instructing the pilots the system was even there. So to the pilots the system acts like a system male-function. There where even such reports. Telling the pilots would possibly need extra training - kind of negating the very reason the system was there.
4) the extra pilot information issued after the 1st crash was not very clear.

So there are 2 mistakes from the technical side and 2 from the instructions / information.

[SiliconWizard]

1) not using redundancy, though redundant hardware is there.
 There was the optional check program to indicate sensor discrepancy - but this did not work in most cases.
2) not checking for non plausible sensor inputs - especially important when using only 1 of the sensors.
3) not telling and instructing the pilots the system was even there. So to the pilots the system acts like a system male-function. There where even such reports. Telling the pilots would possibly need extra training - kind of negating the very reason the system was there.
4) the extra pilot information issued after the 1st crash was not very clear.

Agree with all that. 1) and 2) are basic design mistakes IMO.

Point 3) is interesting. The system was meant to correct a plane design issue. I suppose that openly talking about it in details would have been problematic for Boeing marketing-wise. I personally think this is their main mistake here, and a very severe one. Not the fact they had to implement it. The fact they almost slid that under the carpet.

And point 4): this is indeed completely mind-boggling there was a second occurence.

[Towger]

Boeings killer Planes (2019) Panorama Documentary:

[windsmurf]

Further testing reveal more problems, more delays for Boeing; Cosmic ray "bit-flipping" at high altitudes also considered.
https://www.seattletimes.com/business/boeing-aerospace/newly-stringent-faa-tests-spur-a-fundamental-software-redesign-of-737-max-flight-controls/

[ruffy91]

The text so says that the override switches in the instrument panel are disabled when mcas is active.
Very interesting..

[chris_leyson]

After reading the Seattle Times article I was somewhat surprised to see that single bits are allegedly used to report the in flight status of critical systems. Common sense says that you can't rely on a single bit in a safety crital system. As for cosmic ray bit flipping, unlikely, but it has to be tested, both crashes were at low altitude. A cosmic ray will flip any bit anywhere and anytime, that could apply equally to instructions, status bits, address bits, data bits or any other bit. I think the Seattle Times article is journalism filling in the blanks. Boeing design space hardware so they know about the effect of cosmic rays.

[MT]

Test pilots private conversations on 737 MAX:

[mtdoc]

Prediction:  The 737 Max will not fly again.

I stand by this prediction.

[Nusa]

Prediction:  The 737 Max will not fly again.

I stand by this prediction.

Too bad this isn't a gambling site. I'd take your money betting that it will fly again.

[SilverSolder]

I also think it will fly again, but there is a high risk of delays.  A LOT of work and testing will have to be completely squeaky cleanly completed first...  by the book, dotting the i's and crossing the t's,  by both Boeing and the FAA who are both in the searchlight.   A very bright searchlight - almost more like X-rays than light...  there just isn't going to be any give and take,  perfect is the new official standard.