Lion Air crash: Jakarta Boeing 737 'had prior instrument error'

[KL27x]

Yes, Tooki. This part was a surprise to read. I don't know what to think about this:

When the flight computer trims the airplane to descend, because the MCAS system thinks it’s about to stall, a set of motors and jacks push the pilot’s control columns forward. It turns out that the flight management computer can put a lot of force into that column—indeed, so much force that a human pilot can quickly become exhausted trying to pull the column back, trying to tell the computer that this really, really should not be happening.

Indeed, not letting the pilot regain control by pulling back on the column was an explicit design decision. Because if the pilots could pull up the nose when MCAS said it should go down, why have MCAS at all?

It read the entire thing, but I must have missed the part about the nose gear.

This part was kind of "rewarding" to me, because I've speculated this, myself. But to hear it stated so unequivocally makes me wonder how he knows this is a fact that only fighter jets are dynamically unstable, at all. I imagined there would be degrees to this that could be detectable/demonstrable but still considered safe enough.

Pitch changes with increasing angle of attack, however, are quite another thing. An airplane approaching an aerodynamic stall cannot, under any circumstances, have a tendency to go further into the stall. This is called “dynamic instability,” and the only airplanes that exhibit that characteristic—fighter jets—are also fitted with ejection seats.

no mention of functional safety requirements nor S/W testing and application engineering.

Yeah, I expected something about that from his 40 years of being a software developer. There was no real insight, here, and indeed, he doesn't even demonstrate any software fault of the plane. Design choices, management shortcuts, lack of oversight... not software faults. It is a bit simplified where he suggests that software developers are lazy because they can push patches. Software requires patches for the same reasons he is explaining that failures should be considered a natural part of complex systems. Software can get really complicated, and no human can fully understand all the ramifications of even a moderately complex piece of software. They can understand this bit or that bit. But then there are hundreds of bits all working together, and things just get complicated. He also seems to blame the software developers for doing something well beyond their scope, lol. If Boeing lets a software company fix their plane without specifying the way in which it should be done, and then they don't even check that it meets normal safety criteria, then who is making the mistake?

In absence of any real data, mind you, I would assume that the software engineers made MCAS do exactly what Boeing employees specified, and how they specified it. They might even have made several suggestions for improvements and/or expressed safety concerns that fell on deaf ears, and they just deferred to the experts (and the customer). Even if the software guys were out of their league... if someone hands you a check and asks you to do something you have no experience or expertise in and says, "just figure something out," with no specific parameters, I'd cash the check and figure something out, and I'd detail what exactly the software does. I'd assume it's up to the customer to determine if it is suitable for his purposes or not. Unless my company includes 737 test pilots and I'm given a fleet of test planes and a really big check to cover accidental deaths and insurance for 100 million dollar planes, I'm not really the end judge of these things.

Overall it reads like one of them 60 Minutes old-dude-rants about the good old days. He is obviously painting a narrative. But it was an easy read and included some valid points and insights from a pilot.

[tooki]

Given the shaky factual foundation upon which it’s written, I don’t think any of his “insights” can be given the benefit of the doubt, as far as validity...

[KL27x]

His experience in recertifying his plane after installing an autopilot is interesting. I noticed he had to do a lot of paperwork, mostly. He didn't mention how much this cost. And it came with an extensive manual, but I don't see where he had to take some sort of test. Also, when his friends fly the plane, the only thing they apparently have to do is take a lesson from the owner of the plane and to demonstrate they understand what they need to.

Also in this rant, he has been suggested that if they had disclosed MCAS from the start, the problem would have come from.... the 737 pilots, themselves. Earlier in the history of the 737, I have read that the plane was initially not popular with domestic airlines because pilots and their unions demanded that the plane have a crew of 3 pilots. In other countries, they flew with 2. And this eventually happened in the US after the union capitulated.

It's ironic that, perhaps, the very persons who could use the information is the person you have to hide it from, for cost reasons. 

So all these airline pilots who are saying, "easy, peezy. You just flip the switches, tada, and no problem" is one thing. But the problem as far as the airlines/Boeing was concerned might have been that the pilots union refuses their pilots to fly this new plane until they are promised X hours of training/certification per pilot (at $X per hour).  So this is what happens when politics mixes with safety? The pilots union is perhaps in some way indirectly responsible for the political situation that led to this.

I know from people in the industry stories of mechanics basically not doing their jobs unless incentivized with lucrative overtime hours. But in an orchestrated and overt/open gaming of the system, let's say... like government workers at a feeding trough full of borrowed money with not enough oversight. And airlines going into bankruptcy from this Lord of the Flies atmosphere where the management lost control.

[RandallMcRee]

Given the shaky factual foundation upon which it’s written, I don’t think any of his “insights” can be given the benefit of the doubt, as far as validity...

Hmmm. Think actually you are quite wrong. There is nitpicky stuff which I completely agree with that the article fails (probably) to get right. I think the bigger lesson is correct, its about how to manage and *not* manage life critical projects.

There is a basic engineering truth here--namely the iron triangle.
http://kevinharrisarchitect.com/iron-triangle/
(Stuck in a better link here after original posting).

I was taught this early in my career and have found it to be mostly true.  Boeing picked fast (fixed deadlines) and cheap (totally cost constrained) that means good *had* to suffer. The devil is in the details, the writing was on the wall, etc. etc. As soon as I read in the Seattle times that the two things were paramount--fast&cheap-- the broad strokes of the rest becomes obvious. Boeing management probably picked up some hefty bonuses for managing those d*mn engineers so well. But 347 people died as a result. So, yes, heads need to roll.

[floobydust]

Safety critical software updates do not get released as a "patch" 
There is a formal engineering change order, FMEA assessment, coding, peer review, testing, validation, verification and regulatory assessment. The S/W update goes through an entire approval process. The quality management system (i.e. ISO 9001) is also involved.  A lot of meetings, paperwork and hoops before anything gets released to end users.  There is full traceability with the paper trail.
It easily adds x10 factor for development time compared to basic embedded system software. Many months are needed to properly do a S/W change or add a module.

The need for MCAS might have come very late in the project - during test flights where the aircraft's handling issues arose. This would make it a last minute panic, rush to implement and deploy something.
Keeping MCAS specifications "over-simplified" would be one way to add it quickly and not delay the project.
It's still shocking how Boeing and Collins bungled this software or covered up it's critical nature in order to also skirt giving pilots additional training.
Canada is demanding "pilots should experience the fixes Boeing is devising in simulators" instead of an iPad session, which is against what the FAA is saying.

[Nusa]

Earlier in the history of the 737, I have read that the plane was initially not popular with domestic airlines because pilots and their unions demanded that the plane have a crew of 3 pilots. In other countries, they flew with 2. And this eventually happened in the US after the union capitulated.

It was never a Boeing or FAA requirement, it was about United Airlines union contracts requiring 3-man crews in larger aircraft back in the 1960's, since they were the only US airline flying the 737 in significant numbers at the time. Also, that third man wasn't a pilot, he was a flight engineer. In the 737 he was a glorified secretary. He had a seat with no console and nothing to do other than simple stuff like routine radio traffic and paperwork. It wasn't until about 1972 that UAL got rid of that 3rd man in the cockpit. In commercial aviation, the flight engineer is a near-dead position these days, since most older aircraft that needed them are no longer in service.

So even though you chose to mention it in the same paragraph as MCAS, I rather doubt the the fact there's only been a 2-man crew in the 737 for the last 47 years is relevant in any way. But don't let me stop a good conspiracy theory!

[SkyMaster]

Earlier in the history of the 737, I have read that the plane was initially not popular with domestic airlines because pilots and their unions demanded that the plane have a crew of 3 pilots. In other countries, they flew with 2. And this eventually happened in the US after the union capitulated.

It was never a Boeing or FAA requirement, it was about United Airlines union contracts requiring 3-man crews in larger aircraft back in the 1960's, since they were the only US airline flying the 737 in significant numbers at the time. Also, that third man wasn't a pilot, he was a flight engineer. In the 737 he was a glorified secretary. He had a seat with no console and nothing to do other than simple stuff like routine radio traffic and paperwork. It wasn't until about 1972 that UAL got rid of that 3rd man in the cockpit. In commercial aviation, the flight engineer is a near-dead position these days, since most older aircraft that needed them are no longer in service.

So even though you chose to mention it in the same paragraph as MCAS, I rather doubt the the fact there's only been a 2-man crew in the 737 for the last 47 years is relevant in any way. But don't let me stop a good conspiracy theory!

Are you sure the 737 was operated with a flight engineer? The only seat available in the flight deck is the jump seat, and when deployed it is blocking the flight deck door. The bulkhead with the circuit breakers is right behind the pilots seats.

It was the Boeing 767 that had been designed with a flight engineer position only because at least one union was pushing for this. Unfortunately for the union, the 767 was designed for a crew of two, and the flight engineer had nothing to do.

 

[KL27x]

I rather doubt the the fact there's only been a 2-man crew in the 737 for the last 47 years is relevant in any way. But don't let me stop a good conspiracy theory!

Thanks! Everything else you stated supports the point I made, fantastically. Then there's this nonsensical statement for which I forgive you. 

[GeorgeOfTheJungle]

[soldar]

New York Times: Claims of Shoddy Production Draw Scrutiny to a Second Boeing Jet. Workers at a 787 Dreamliner plant in South Carolina have complained of defective manufacturing, debris left on planes and pressure to not report violations.

At the North Charleston plant, the current and former workers describe a losing battle with debris.

That describes the situation I have in my workroom: " a losing battle with debris".

[GeorgeOfTheJungle]

https://www.nytimes.com/2019/03/14/business/automated-planes.html

In nearly 100 million flights by United States passenger airlines over the past decade, there has been a single fatality. Other than most landings and takeoffs, the planes have largely been flying themselves.

But the recent crashes of Boeing 737 Max 8 jets in Indonesia and Ethiopia have raised questions about the downside of all that automation.

Pilots now spend more time learning these automated systems than practicing hands-on flying, so newer pilots are less comfortable with taking manual control when the computer steers them wrong, according to interviews with a dozen pilots and pilot instructors at major airlines and aviation universities around the world.

“The automation in the aircraft, whether it’s a Boeing or an Airbus, has lulled us into a sense of security and safety,” said Kevin Hiatt, a former Delta Air Lines pilot who later ran flight safety for JetBlue. Pilots now rely on autopilot so often, “they become a systems operator rather than a stick-and-rudder pilot.”
As a result, he said, “they may not exactly know or recognize quickly enough what is happening to the aircraft, and by the time they figure it out, it may be too late.”

In October, a Lion Air jet crashed in Indonesia, killing 189 people. Investigators now think the pilots struggled to control the Boeing aircraft after its automated systems malfunctioned, in part because they didn’t fully understand how the automation worked. The authorities are investigating what caused Sunday’s crash of the same model jet in Ethiopia, in which 157 people died.
While automation has contributed to the airline industry’s stellar safety record in recent years, it has also been a factor in many of the crashes that have still occurred around the world. A 2011 study by a federal task force found that in about 60 percent of 46 recent accidents, pilots had trouble manually flying the plane or handling the automated controls. Complicated automation systems can also confuse pilots and potentially cause them to take action they shouldn’t, pilots said.

President Trump weighed in on Tuesday, posting on Twitter that airplanes have become too technologically complex and that he wants “great flying professionals that are allowed to easily and quickly take control of a plane!”

https://thepointsguy.com/news/captain-sully-sullenberger-on-737-max-crash/

We do not yet know what caused the tragic crash of Ethiopian 302 that sadly claimed the lives of all passengers and crew, though there are many similarities between this flight and Lion Air 610, in which the design of the Boeing 737 MAX 8 is a factor. It has been obvious since the Lion Air crash that a redesign of the 737 MAX 8 has been urgently needed, yet has still not been done, and the announced proposed fixes do not go far enough. I feel sure that the Ethiopian crew would have tried to do everything they were able to do to avoid the accident. It has been reported that the first officer on that flight had only 200 hours of flight experience, a small fraction of the minimum in the U.S., and an absurdly low amount for someone in the cockpit of a jet airliner. We do not yet know what challenges the pilots faced or what they were able to do, but everyone who is entrusted with the lives of passengers and crew by being in a pilot seat of an airliner must be armed with the knowledge, skill, experience, and judgment to be able to handle the unexpected and be the absolute master of the aircraft and all its systems, and of the situation. A cockpit crew must be a team of experts, not a captain and an apprentice. In extreme emergencies, when there is not time for discussion or for the captain to direct every action of the first officer, pilots must be able to intuitively know what to do to work together. They must be able to collaborate wordlessly. Someone with only 200 hours would not know how to do that or even to do that. Someone with that low amount of time would have only flown in a closely supervised, sterile training environment, not the challenging and often ambiguous real world of operational flying, would likely never have experienced a serious aircraft malfunction, would have seen only one cycle of the seasons of the year as a pilot, one spring with gusty crosswinds, one summer of thunderstorms. If they had learned to fly in a fair-weather clime, they might not even have flown in a cloud. Airlines have a corporate obligation not to put pilots in that position of great responsibility before they are able to be fully ready. While we don’t know what role, if any, pilot experience played in this most recent tragedy, it should always remain a top priority at every airline. Everyone who flies depends upon it

https://www.boeing.com/commercial/737max/737-max-software-updates.page

[Nusa]

https://www.nytimes.com/2019/03/14/business/automated-planes.html

In nearly 100 million flights by United States passenger airlines over the past decade, there has been a single fatality.

That statistic hasn't changed, since everything the rest of the article talks about were not United States passenger airlines, nor did they happen in the United States. Plus it didn't even mention that the single fatality had nothing to do with crashing or automation. Crappy journalism leading with that at all, since that's not what they wanted to talk about.

[soldar]

>> "Investigators now think the pilots struggled to control the Boeing aircraft after its automated systems malfunctioned, in part because they didn’t fully understand how the automation worked."

And the reason they didn't fully understand how the automation worked was because Boeing sold it as something that did not need to be understood and required no extra training. Because, you know, training costs time and money which could get in the way of sales.

So, basically, it is the fault of the pilots ... for trusting Boeing.

[GeorgeOfTheJungle]

So, basically, it is the fault of the pilots ... for trusting Boeing.

Boeing and pilots that crash it the minute something fails too.

[soldar]

The pilots had no training for that scenario and the reason they had no training for that situation was that Boeing said no training was needed.

Who is responsible for the pilots not having been trained for that situation?

[KL27x]

A cockpit crew must be a team of experts, not a captain and an apprentice. In extreme emergencies, when there is not time for discussion or for the captain to direct every action of the first officer, pilots must be able to intuitively know what to do to work together. They must be able to collaborate wordlessly. Someone with only 200 hours would not know how to do that or even to do that.

If real pilots are trained the way Mentour Pilot simulations go... boy, I think there's a conflict, here.
Capt:     The plane appears to be crashing. Agree?
Copilot:  Yes, I concur. Plane appears to be crashing.
Capt:      Proceeding to memory items for crashing of plane. Please. Perform memory items for crashing plane.
Copilot:  Memory items for crashing plane. Step 1, put head in ass. Permission to put head in ass?
Capt:     Yes, please proceed with head in ass.

 
 

It has been reported that the first officer on that flight had only 200 hours of flight experience, a small fraction of the minimum in the U.S., and an absurdly low amount for someone in the cockpit of a jet airliner... A cockpit crew must be a team of experts, not a captain and an apprentice.

I imagine this is important if the captain is incapacitated. But failing that, I hope a modern plane can be flown with one competent pilot and one "assistant" who can take orders and perform medial tasks as directed. Whose main reason d'etre is for redundancy/backup.

[GeorgeOfTheJungle]

The pilots had no training for that scenario and the reason they had no training for that situation was that Boeing said no training was needed.

They should have known perfectly well how to handle that issue since november 6 2018, because "Following last week’s crash of a brand-new Lion Air Boeing 737 MAX 8 in Indonesia, the plane’s manufacturer has issued an emergency Airworthiness Directive, warning all the aircraft’s operators of a potential instrument failure that could force the plane to fly into a steep dive" and the FAA another:

https://duckduckgo.com/?q=737+emergency+directive+trim+runaway

[KL27x]

They should have known perfectly well how to handle that issue since november 6 2018, because "Following last week’s crash of a brand-new Lion Air Boeing 737 MAX 8 in Indonesia, the plane’s manufacturer has issued an emergency Airworthiness Directive, warning all the aircraft’s operators of a potential instrument failure that could force the plane to fly into a steep dive" and the FAA another:

Yeah, that's why the crash was such a big deal. The pilots WERE aware, but they still lost control of the plane.

I can't remember where I read this. Initially, the "roller coaster maneuver" was described in the operating manual of the 737. It was recommended to lift the nose as much as possible, then push the nose down while moving the trim. And repeating as many times as necessary. But later this was removed. There was still a warning that the stabilizer could potentially get stuck, but this was moved to a supplemental manual. And the description of how to handle this was completely deleted. It is unclear if this is still taught/trained to pilots or if it got relegated into obscurity, since the need never arose... until MCAS. 

Then effect of speed was perhaps not obvious, either. The AD appears to cover the cutting out of the power trim after recognizing MCAS activation, but not including what you may have to do afterward.

[GeorgeOfTheJungle]

But they did NOT need to do that (roller coaster maneuver), for the umpteenth time: they just had to (un)trim electrically and flip the cutout switch after that. Any Sully would have done that instinctively, immediately, in a sec, end of the problem.

[soldar]

So the planes were all grounded, worldwide, for no good reason except that all pilots, worldwide, are incompetent. I think I get it now.

[KL27x]

GoJ, You still don't know that, though, do you? The prelim that Ethiopian Airlines released does not include the duration of trim inputs from the pilots. Did I miss something?

We have the info for the Jakarta flight recorders. The trim input was too little at the end. I have not seen this for ET302.

Also, why do you assume that the pilots would know that under-correcting a mistrim is a fatal error? In a world where planes fly themselves, the rule is to not do anything wrong. If the plane didn't crash yet, and you made it a bit better, you might think it will still not crash.

If you make it better, and it is better at first... but it can progressively gets worse very quickly in an exponential way... that is something that you might want to know in advance.

Any Sully would have done that instinctively, immediately, in a sec, end of the problem.

Apparently, Sully is no longer allowed to fly planes. Now the capt and copilot must verbally go through a checklist and announce their intentions and ask permissions before flipping switches and pressing buttons. This is at least what they would do if they are not aware of how fast this can go south. They may have thought the worst was over.

Unless you can clone Capt Sully and include one with every plane, that might not be a very good solution.

Just to be clear, and maybe I'm undertanding things incorrectly, the AD stresses MCAS as the issue and that the solution is to cut stab trim and then MANUALLY trim the plane. Furthermore, at a given initial speed, simply pulling up on the yoke is enough to correct the plane. So if you pull on the yoke, and the plane flies how you want it to, you think the emergency is over. You don't have to correct the trim ALL THE WAY, like you are saying would be automatic, 1 second, done by Sully. (ironically, it might take several long seconds with power). You've cut stab trim; let's further say you have disabled autothrottle. The plane is level, because you are pulling on the yoke. You are still in imminent danger... because of a level of MISTRIM of which pilots apparently do not much relevant experience and training... not because of the potential for further MCAS error.

[GeorgeOfTheJungle]

So the planes were all grounded, worldwide, for no good reason except that all pilots, worldwide, are incompetent. I think I get it now.

Where in Spain do you come from? Your logic is flawed.

[ogden]

So the planes were all grounded, worldwide, for no good reason except that all pilots, worldwide, are incompetent. I think I get it now.

Where in Spain do you come from? Your logic is flawed.

Where in Poland do you come from? Your perception of sarcasm is flawed.

[GeorgeOfTheJungle]

GoJ, You still don't know that, though, do you? The prelim that Ethiopian Airlines released does not include the duration of trim inputs from the pilots. Did I miss something?

If it's true that the (electric) trim button overrides the MCAS trim commands it follows that they just had to keep pushing it longer. The emergency directive quite clearly suggests that too:

Initially, higher control forces may be needed to overcome any stabilizer nose down trim already applied. Electric stabilizer trim can be used to neutralize control column pitch forces before moving the STAB TRIM CUTOUT switches to CUTOUT.

[ogden]

If it's true that the (electric) trim button overrides the MCAS trim commands it follows that they just had to keep pushing it longer.

Yes, it is true that button overrides MCAS. It is clearly visible in the flight recorder data. Did they knew/notice or not - that's the question which is unanswered.

[edit] Illustration attached. Notice slightly shorter one of three MCAS long trims - because aborted by manual input. Those two graphs tells whole story of the tragedy - cause and possible solution as well.