Lion Air crash: Jakarta Boeing 737 'had prior instrument error'

[Simon]

In one of the "Air Crash Investigation" episodes one of the investigators said they internationally do not watch any kind of news before examining the crash site

Yes the fake news media would pollute any investigation.

Ordinary trials in the UK have been thrown out because just 1 member of the jury decided to go detectiving for themselves rather than rely solely on the evidence presented in court.

[GeorgeOfTheJungle]

From the video at #376:
"[...] The software fix is going to require this flight control system to rely on data from TWO sensors, it previously was only dependent on one [...]"

Really? The MCAS is reading only one sensor? That seems unbelievable to me. The engineers at Boeing no more no less, should have known better.

[Simon]

safty critical with one sensor? ouch! idealy you want 3 so you know who to trust.

[GeorgeOfTheJungle]

safty critical with one sensor? ouch! idealy you want 3 so you know who to trust.

Even the accelerator pedal of my 1998 ford focus has three separate potentiometers!

Is this (in bold, below) a weasel-ish way of saying "from now on we're going to read the second AoA sensor too"?

Boeing Statement On 737 MAX Software Enhancement
http://aero-news.net/index.cfm?do=main.textpost&ID=78526990-A846-4929-8D11-9D3C0FDD9DE3

For the past several months and in the aftermath of Lion Air Flight 610, Boeing has been developing a flight control software enhancement for the 737 MAX, designed to make an already safe aircraft even safer. This includes updates to the Maneuvering Characteristics Augmentation System (MCAS) flight control law, pilot displays, operation manuals and crew training. The enhanced flight control law incorporates angle of attack (AOA) inputs, limits stabilizer trim commands in response to an erroneous angle of attack reading, and provides a limit to the stabilizer command in order to retain elevator authority.
[...]

[donotdespisethesnake]

From the video at #376:
"[...] The software fix is going to require this flight control system to rely on data from TWO sensors, it previously was only dependent on one [...]"

Really? The MCAS is reading only one sensor? That seems unbelievable to me. The engineers at Boeing no more no less, should have known better.

I think the idea will be to cross-check the two sensors for agreement, if they disagree then disable MCAS and issue a warning to the pilots.

While the engineers at Boeing should (and probably do) know better, they are overruled by finance and marketing these days.

[GeorgeOfTheJungle]

I think the idea will be to cross-check the two sensors for agreement, if they disagree then disable MCAS and issue a warning to the pilots.

From reply #339:

•   IAS DISAGREE alert.
•   ALT DISAGREE alert.
•   AOA DISAGREE alert (if the option is installed).
•   FEEL DIFF PRESS light.

AoA DISAGREE is (was?) an option! 

But then, the second AoA sensor what for was there? For decoration?

[donotdespisethesnake]

I think the idea will be to cross-check the two sensors for agreement, if they disagree then disable MCAS and issue a warning to the pilots.

From reply #339:

•   IAS DISAGREE alert.
•   ALT DISAGREE alert.
•   AOA DISAGREE alert (if the option is installed).
•   FEEL DIFF PRESS light.

AoA DISAGREE is (was?) an option! 

But then, the second AoA what for was there? For decoration?

The second sensor is redundant, I think the sensors are fed to each seat, similar to how other controls such as altimeter. I.e. captain seat gets sensor input from LH sensors, FO seat gets info from RH sensors. The normal CRM would have the pilots determine the faulty sensor and ignore it. Unfortunately CRM is often lacking in upset situations. In this case, if they don't quickly identify the need to disable electric trim it is too late.

The AoA DISAGREE is an option few airlines have taken. It's up to the pilots to see it then disable electric trim (which also prevents MCAS moving the stabilizer).

I think the MCAS can currently only take one AoA or the other, so the fix would be to use both, or at least use the AoA DISAGREE signal to automatically disable MCAS, presumably there needs to be another alert to indicate MCAS Inoperative.

I'm afraid to say it's a set of bodges to cater for the original decision to add bigger engines, AND retain 737 flight characteristics so that minimal type conversion is required for the pilots.

Unfortunately, I see the same trend in other industries. Marketing, finance are allowed to override proper engineering. The question is always "how much will it cost?", and not "is the quality right?"

[KL27x]

^ I don't get how this is even a cost issue. It seems much more like an oversight.

[donotdespisethesnake]

^ I don't get how this is even a cost issue. It seems much more like an oversight.

Because the 737 MAX grandfathers certification and type training from the 737 Classic. If Boeing change the design significantly, it needs a whole new certification - expensive and time consuming. If the flight characteristics change, it means new type training for the pilots, so operators need to be spend money on training, and the pilots need to get type certificate.

Boeing wanted to get a plane into production quickly, because Airbus was outselling them by a wide margin.

By telling regulators and operators that "the MAX can be treated the same as the classic", Boeing took a big shortcut compared to designing a new airframe. It seems that gamble created a major flaw.

[floobydust]

It's a Boeing design error and the S/W patch allowing "the option" of using two sensors stinks of cover-up. AOA DISAGREE alert indicator is also an "option"?! 

Using two sensors is still shitty because you have now doubled the probability of MCAS failure due to a sensor failure.
Multiple sensors iced up for the AF447 disaster. Would MCAS register a discrepancy with two sensors reading similar yet both are out to lunch?
I'll repeat the old adage "with two clocks you can never know the correct time". This MCAS system is never going to be stellar, even adding a third (sensor) opinion because the other pair can malfunction. It's just getting a slightly lower probability of failure, this is all Boeing can accomplish. Unless there was a gross S/W bug that is being fixed too.

In other industries with safety-critical design, you do fault-tree analysis and FMEDA to ensure you have coverage of a sensor problem, among other scenarios.
Clearly, Boeing bungled this and is showing a repeat bungle with their hasty "software fix" that cannot meet basic functional safety requirements even after piling on the algorithm smartness.
I've seen this before - a bad design safety-critical system is out there, sold in numbers and a corporation has a massive panic to fix it ASAP without changing any hardware.
Adding complex S/W algorithms (which can never be proven correct) is very dangerous.

Then I read this:
"MCAS is implemented within the two Flight Control Computers (FCCs). The Left FCC uses the Left AOA sensor for MCAS and the Right FCC uses the Right AOA sensor for MCAS. Only one FCC operates at a time to provide MCAS commands. With electrical power to the FCCs maintained, the unit that provides MCAS changes between flights. In this manner, the AOA sensor that is used for MCAS changes with each flight."

How do you come up with something so stupid?

[Simon]

I assumed that for a safety critical system you measure everything with 3 sensors, should one fail you can identify it is the failed one as two measurements still agree and you flaf the faulty sensor.

[MT]

On the horizon i can see a large number of relatives preparing to sue Boing.

Boeing have been working on a software modification to MCAS since the Lion Air accident. Unfortunately although originally due for release in January it has still not been released due to both engineering challenges and differences of opinion among some federal and company safety experts over how extensive the changes should be. Apparently there have been discussions about potentially adding enhanced pilot training and possibly mandatory cockpit alerts to the package. There also has been consideration of more-sweeping design changes that would prevent faulty signals from a single sensor from touching off the automated stall-prevention system.

[floobydust]

Somebody remind me why MCAS is even needed?
This 737 max 8 is presumably empty, but none the less, wow an impressive climb.

ET302 took off like a bat out of hell, hard vertical climb for the first few seconds after takeoff so I wonder if a sensor was not working right from the start.

[chickenHeadKnob]

Somebody remind me why MCAS is even needed?
This 737 max 8 is presumably empty, but none the less, wow an impressive climb.

ET302 took off like a bat out of hell, hard vertical climb for the first few seconds after takeoff so I wonder if a sensor was not working right from the start.

The angle of attack sensors don't measure the angle with the ground or horizon, rather they measure the relative angle of the airflow to the center line of the aircraft. So if the aircraft is moving fast enough and has a high enough thrust to weight ratio it can keep on accelerating even in a steep climb. If the airspeed is increasing and the climb angle stays constant the angle of attack will actually decrease.

None the less I bet the factory pilots rehearse that manoeuvre and probably disabled MCAS.  They likely also had in mind the Airbus 320 Demo flight crash where the pilots where in inadvertently fighting HAL 9000.

[Nusa]

On the horizon i can see a large number of relatives preparing to sue Boing.

Boeing have been working on a software modification to MCAS since the Lion Air accident. Unfortunately although originally due for release in January it has still not been released due to both engineering challenges and differences of opinion among some federal and company safety experts over how extensive the changes should be. Apparently there have been discussions about potentially adding enhanced pilot training and possibly mandatory cockpit alerts to the package. There also has been consideration of more-sweeping design changes that would prevent faulty signals from a single sensor from touching off the automated stall-prevention system.

The FAA people involved in the process would have been unavailable from December 22 to January 25. Trump personally added a month to the schedule as part of the unintended consequences of his budget shutdown. Boeing can't release squat without FAA approval. Nor can they they resolve any engineering differences when the FAA isn't available. So Boeing should get a pass on at least a month of that delay.

[djacobow]

Somebody remind me why MCAS is even needed?
This 737 max 8 is presumably empty, but none the less, wow an impressive climb.

Actually, the MCAS was needed in part *because* of those powerful engines. They are more powerful and mounted farther forward than the engines on other 737s, so their power generates a higher turning moment that wants to push the nose up, so the danger of a stall is increased. Furthermore, once you the AOA very high, the nacelles themselves generate some lift (again, with a large moment because of the forward placement relative to the center of rotation), and that pushes for an even higher AOA. This is really only a problem once you are already at high AOA, but essentially the engines themselves become a destabilizing force once you get out of the safe zone.

MCAS is designed to keep the scenario from running away.

Pretty good explanation of why you don't add power immediately after a stall here:

Pretty good explanation of why high AOA's are more problematic in the MAX here: https://leehamnews.com/2018/11/14/boeings-automatic-trim-for-the-737-max-was-not-disclosed-to-the-pilots/

[Gyro]

Hmmm, at what point does the Max become a different plane? When you start moving the engines around, changing its dynamics etc...

[mtdoc]

Read this Twitter thread for an interesting take on the root causes.  Fuel prices/economic considerations led to a cascde of band aid design and sytems changes each one an attempt to compensate for problems introduced by the prior. The “software patch” is just the latest.

I won’t be surpised to see more of these kind of economics/complexity/systems problems in socierty going forward as there is more and more pressure on engineers to provide technology fixes to underlying economics/resource problems.

[djacobow]

Read this Twitter thread for an interesting take on the root causes.  Fuel prices/economic considerations led to a cascde of band aid design and sytems changes each one an attempt to compensate for problems introduced by the prior. The “software patch” is just the latest.

I won’t be surpised to see more of these kind of economics/complexity/systems problems in socierty going forward as there is more and more pressure on engineers to provide technology fixes to underlying economics/resource problems.

Providing technology fixes to economic problems is pretty much what engineering is. I think "An engineer can do for a dollar what any fool can do for two" is attributed to Arthur Wellington, a well known civil engineer from the late 1800's.

I don't really see the problem with Boeing setting the design goal for the MAX to improve performance without requiring a new type certificate. That the MCAS may be a bad implementation, or the specific engine placement requiring MCAS was a bad idea, or the whole thing has led to complex human+automation failure modes is a sign that maybe they screwed up.

But it is one thing to say that an engineering project failed, quite another to state as SO many have done, IMHO, talking without much knowledge, that the concept itself was doomed to fail. I suspect that in reality, Boeing will implement some fixes and there will be some training, and the MAX will ultimately become a safe airplane like the rest of the 737's.

[Bud]

It turns out i personally know someone who lost his family members on the Ethiopian flight.... Cant imagine what that person is going through...

[GeorgeOfTheJungle]

I suspect that in reality, Boeing will implement some fixes and there will be some training, and the MAX will ultimately become a safe airplane like the rest of the 737's.

Yes it will, I think so too, but with the MCAS... they royally screwed up.

[KL27x]

Actually, the MCAS was needed in part *because* of those powerful engines. They are more powerful and mounted farther forward than the engines on other 737s, so their power generates a higher turning moment that wants to push the nose up, so the danger of a stall is increased. Furthermore, once you the AOA very high, the nacelles themselves generate some lift (again, with a large moment because of the forward placement relative to the center of rotation), and that pushes for an even higher AOA. This is really only a problem once you are already at high AOA, but essentially the engines themselves become a destabilizing force once you get out of the safe zone.

I'm struggling with this description. It makes it seem like the new engine placement has a tendency to push the nose of the plane up, and that is the main reason for MCAS.

From the pieces I have gathered, the problem requiring the MCAS is that the engine size and placement altered the aerodymanics of the plane in a way that made it unstable. It's not just a matter of the pilot needed to adjust the trim to adjust for a predictable and steady nose-up tendency. It sounds like the plane becomes unstable in a way that the trim would have to be whacked around up and down in rather large and fast adjustments to make the ride smoother.

In addition to this link posted by mtdoc, I have read another article that suggested the "ride comfort" was adversely affected the the change in engine position, and the MCAS was added to make the flight smoother. But, alas, I have no link.

Read this Twitter thread for an interesting take on the root causes. 

Trevor Sumner suggests that the software was never the problem and worked fine. That it was a bandaid that was necessary because of the change in engine placement and the resulting change in aerodynamics. But if anything, it sounds like the software bandaid was indeed part of what turned a "bumpy ride smoother-outer" into a death trap. Yes, the pilots should have cut out the stab trim or what not. And I don't know why there were never informed of the deadly potential problem with the MCAS. But it would have been very trivial (in hindsight) to make MCAS unable to override manual input to such a sustained degree. If it was there to make the ride smoother, then it could have been implemented (thru software) to be able to improve the ride without killing the passengers. Given that a sensor malfunction could have theoretically happened out of the blue at least as low as 6,000 feet and requiring manually turning off the system and course correction within just mere seconds (flipping multiple switches on the roof of the cabin while plummeting towards the earth in near free fall, then cranking the trim wheel bit by bit), it might have happened even with a prepared pilot that knew about the potential fault in advance.

In the light of what I'm inferring, the MCAS system would also have to account for resonant frequencies/oscillations, in addition to sensor malfunction. There would have to be some smarts to it, lets say. Not just a simple if/then cause and effect.

[TheSteve]

Are the new engines so powerful that MCAS is needed or is it purely a bandaid because the weight of the new engines changed the CG too much and they were too cheap to re-locate the wings to the proper place?

[KL27x]

^that's what I am wondering.

AFAIK, the engines were too big for ground clearance. So to use the existing 737 frame, they had to change the mounting position.

Due to Boeing's philosophy that the plane should be able to fly with only manual control, I assume this means the new engines and mounting position are still safe and stable enough to be flown, manually... but I have inferred (right or wrong) that the ride would be less smooth/comfortable without MCAS.

[chickenHeadKnob]

I'm struggling with this description. It makes it seem like the new engine placement has a tendency to push the nose of the plane up, and that is the main reason for MCAS.

That is exactly the reason for MCAS ; safety, not ride comfort. Youtuber Mentour pilot, who is a 737NG captain explains this.

For powered aircraft there is a corner in the flight envelope at high angles of attack called "getting behind the power curve". Aerodynamically past a certain high angle of attack,  before full stall the sink rate/drag on the aircraft increases dramatically and if power is applied with low slung engines the pitch up moment makes things worse.