Another deadly 737 Max control bug just found!

[windsmurf]

American Airlines and United Airlines extend 737 Max grounding through at least early November

https://www.latimes.com/business/la-fi-boeing-max-american-united-grounding-20190714-story.html

[floobydust]

That's 8,000 flights for United Airlines $185M revenue. I say the 737 max won't be flying until well into 2020.
I wonder if the money and lives lost will ever be a motivator to do it right, or will it be just investor's cash being burned for a little "hiccup".

Instead of parking lots full, maybe Boeing could turn the planes into condos? Just park under the wings.

[aix]

Looks like there's a rebranding exercise going on: https://www.bbc.co.uk/news/business-48995509

[Gyro]

Not sure if it's a knock-on but Ryanair are now saying that they will have to cut flights and possibly bases next year as a result of the delay...

https://www.bbc.co.uk/news/business-49000796

It would be amazing if they were actually trying to get away with rebranding the Max. Ryanair have a pretty poor reputation.

[splin]

And why machines for which the state cannot be documented due to things like heap allocation should not be used in safety critical applications.  This also makes processor features which contain unknown state like caches, speculative execution, and multi-threading less desirable.

Except that pretty much any non-trivial system will use heap allocation - but it will be called something else, typically a buffer pool or the like. Try, for example, implementing a comms protocol without one. These pools will be safer than a global heap because (at least):

a) They typically will use fixed size allocations (or a limited number of fixed sizes from different pools) so should be free from heap fragmentation which is one of the bigger problems with heaps.

b) They will also be shared between a limited subset of the whole application.

Typically this managed memory will be passed between tasks/processes/subsystems - eg. between different layers in a comms stack, including device drivers, with all the attendant risks due to the distance and time seperating the allocator of the memory, the users and the consumer that has to de-allocate the memory.

By distance, I mean between developers, who may be in different teams, requiring coordination/documentation and suitable development tools to try to ensure that problems such as memory leaks, accessing free'd off memory etc. are minimized (I'd say eliminated if that were possible).

You can improve things by copying buffers between tasks/subsystems if you can afford the performance and memory cost - but how many systems have the luxury? If you have finite resources someone has to decide how to allocate it. More stack or more buffer space? In the case of Toyota, they didn't allocate enough for stack space but had they increased the stack allocation something else would have less memory available - ok if you can determine in advance the worst case memory usage but there are many times when static analysis tools can't be used and it comes down to the skill of the developers to work out the worst cases.

For safety critical applications you obviously must be able to guarantee the behaviour of the critical parts of the system and isolate them from less trusted subsystems. The reality is that tradeoffs between cost, development time/effort (re-certifying a 737 MAX), functionality and safety are always being made - there rarely absolutes - if ever. No point in a seven-nines reliability requirement if it costs ten trillion dollars. Or a car ECU which costs $1500 and can't do MP3. An aeroplane with enough redundancy to be guaranteed never to fail is probably too heavy to fly. Software that takes longer to develop than the life cycle of the product it's used on is pointless.

[Rick Law]

...
...
 Software that takes longer to develop than the life cycle of the product it's used on is pointless.

I think software problem is more because of it being "lower entry barrier."  Anyone with enough money to buy a cheap laptop can start developing software.  Compare the cost of starting a new software development company to say cost of starting a company to develop a new video chip to compete with NVIDIA.  The ratio of the two cost-of-entry would (with exaggeration) overflow a typical calculator.

This low entry cost of software development leads to the average quality of the developer, quality of the development shop, experience of the development shop all correspondingly lowered.  You could, in theory, have people writing software for a washing machine who has never seen a working washing machine, and they are writing their washing machine software in a shed next to a river bank next to the bunch of people washing their cloths on the river shore.  You could do software in that environment, but you are not going to develop any new IC's of complexity sitting in shed with bathrooms in the out-house.

Yeah, I exaggerated.  I exaggerate to point out how the lower entry barrier can bring in much lower caliber teams.

...
I wonder if the money and lives lost will ever be a motivator to do it right, or will it be just investor's cash being burned for a little "hiccup".

Instead of parking lots full, maybe Boeing could turn the planes into condos? Just park under the wings.

They will remember it for a while, sack some of the ones involved (who merely followed order too do everything needed to reduce cost), and bring in new blood.  Hiring these new blood will be announced as investment to make sure such problem will be fixed and make sure such fiasco will never happen again.  As their stock improves (even if the stock price increase is due to improvement in the general economy or due to inflation), their memory fade as the stock price rises...

The incoming new blood will of course hire more new guys.  Most important of the new hires will be the ones for the new marketing team - the team that will do magic to wash off the stain of "sins of prior management team" in the public's (and the customer's) eyes.  But, the new marketing drive needs funding.  So, budget cut for the development groups to fund the new marketing drives.  They did a bad job that caused the problem, so they deserve to have their budget cut to the bone and then some.  They will be told to find ways to do software development "smarter".  That is to say, do it faster, better, and with even less money.  If the development team(s) can't do that, we fire the whole lot and outsource the whole darn thing.

[GeorgeOfTheJungle]

You could do software in that environment, but you are not going to develop any new IC's of complexity sitting in shed with bathrooms in the out-house.

Look: youtube.com/watch?v=QqxThgLTLyk&t=7m54s :-)

Part one: youtube.com/watch?v=jhwwrSaHdh8 and part two: youtube.com/watch?v=re5xAqgKqc0

[David Hess]

And why machines for which the state cannot be documented due to things like heap allocation should not be used in safety critical applications.  This also makes processor features which contain unknown state like caches, speculative execution, and multi-threading less desirable.

Except that pretty much any non-trivial system will use heap allocation - but it will be called something else, typically a buffer pool or the like. Try, for example, implementing a comms protocol without one. These pools will be safer than a global heap because (at least):

...

How complex does a system involving physical control loops need to be?  Separate it and stick it on redundant hardware.

Static allocations are wasteful but so what?  Memory is cheap.

For safety critical applications you obviously must be able to guarantee the behaviour of the critical parts of the system and isolate them from less trusted subsystems. The reality is that tradeoffs between cost, development time/effort (re-certifying a 737 MAX), functionality and safety are always being made - there rarely absolutes - if ever. No point in a seven-nines reliability requirement if it costs ten trillion dollars. Or a car ECU which costs $1500 and can't do MP3. An aeroplane with enough redundancy to be guaranteed never to fail is probably too heavy to fly. Software that takes longer to develop than the life cycle of the product it's used on is pointless.

I suspect the larger problem is that the programming techniques are no longer taught and the tools are no longer available.

In this case none of that mattered because the system operated exactly as designed.  It was just designed ineptly.

[coppercone2]

there is a obsession with memory space in conventional software design that should not be carried over to embedded systems. Even with code clarity.

[windsmurf]

Speaking of memory space, memory overflow is forcing Airbus planes to power cycle the entire avionics suite every 149 hours

https://www.theregister.co.uk/2019/07/25/a350_power_cycle_software_bug_149_hours/

[David Hess]

That reminds me of the accumulation of residual errors causing problems with the Patriot missile system:

https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dhahran

[BravoV]

Speaking of memory space, memory overflow is forcing Airbus planes to power cycle the entire avionics suite every 149 hours

https://www.theregister.co.uk/2019/07/25/a350_power_cycle_software_bug_149_hours/

I guess Windows 95 or 98 are much better as they only need to be restarted at every 49.7 days. 

[GeorgeOfTheJungle]

(2^32)/1000/3600/24= 49.7

[Kleinstein]

there is a obsession with memory space in conventional software design that should not be carried over to embedded systems. Even with code clarity.

With modern PC software they don't seem to care abut memory anymore. Software gets bloated so much it no longer runs on older system and update take forever with a slow internet connection. Also the feeling is that software quality tends to go down the larger a program - kind of normal with the time needed for debugging going up usually more like the 3rd power or even exponential with code length.

Software development can be quite different for a modern multi tasking OS, an old PC and embedded systems with OS (e.g. RTOS) and embedded without an OS. Especially for a critical systems things like static memory allocations and maybe not using pointers at all can be a good solution. Dynamic memory can cause odd effects, like fragmented memory or a garbage collector to sometimes take a long time.

Those programmers that started with core rope, mask-ROMs and one time programmable µCs now are retiring.

[David Hess]

We found an bug in the IP stack of Windows 2000 which slowly leaked memory requiring a reboot about every 2 weeks on a low memory system.  Microsoft's only recommendation was to install more memory and they never fixed it.

[SiliconWizard]

We found an bug in the IP stack of Windows 2000 which slowly leaked memory requiring a reboot about every 2 weeks on a low memory system.  Microsoft's only recommendation was to install more memory and they never fixed it.

Is that what we call being "practical"?

[SiliconWizard]

How complex does a system involving physical control loops need to be?  Separate it and stick it on redundant hardware.

Static allocations are wasteful but so what?  Memory is cheap.

Yeah I agree. Control loops in avionics are still based on pretty simple algorithms. No need for any dynamic allocation.

MISRA-C is clear about that.
https://www.misra.org.uk/forum/viewtopic.php?t=1233

I believe the rules are, or at least should reasonably be the same in avionics software.

If using dynamic allocation, you would at least have to justify that thoroughly. So I'm wondering what kind of arguments could be used here.
I know some teams heavily rely on automated tools (such as dynamic code analysis), so that if the tools don't catch any potential memory leak, the team considers the code OK. That gives them a feeling of a safety net and gives them the possibility of using dynamic allocation without having to think twice about it (thinking the tools will catch all problems). This is bad.

On a general level, following no or fewer rules just because there are machines or people that are going to check an engineer's work is a recipe for bad design...

[StillTrying]

UK TV tomorrow night.

Boeing's Killer Planes
Panorama
The Boeing 737 Max was set to dominate the skies, but two deadly crashes have left the aircraft grounded and the company in crisis. Both flights were forced down by software specially designed for the new aircraft. Reporter Richard Bilton investigates the plane’s fatal flaws and asks whether Boeing should have done more to protect passengers.
https://www.bbc.co.uk/programmes/m00077cw

[MT]

How did that tittle even pass BBC program board executives?   BBC seams desperate these days yet again connected to pedophilia and all!

[Towger]

The BBC culture of old appears to have been left behind in London when it sold up and moved to Manchester.

I am sure Trump will have an interesting tweet about the program...

[CiscERsang]

The BBC culture of old appears to have been left behind in London when it sold up and moved to Manchester.

I am sure Trump will have an interesting tweet about the program...

His team is already looking for variants on helping to Boeing corporation to overcome the crisis, I believe.

[Towger]

A former Boeing engineer has told the BBC's Panorama programme that work on the production line of the 737 Max plane was not adequately funded.

https://www.bbc.com/news/business-49142761

[CiscERsang]

A former Boeing engineer has told the BBC's Panorama programme that work on the production line of the 737 Max plane was not adequately funded.

https://www.bbc.com/news/business-49142761

  Shoked by details revealed. That's deadly side effect of capitalism.

[SiliconWizard]

Well I dunno. "Killer planes" seems like an appropriate pun here. Sure at Boeing they are not going to like the humor.

[MyHeadHz]

A former Boeing engineer has told the BBC's Panorama programme that work on the production line of the 737 Max plane was not adequately funded.

https://www.bbc.com/news/business-49142761

  Shoked by details revealed. That's deadly side effect of capitalism.

It can be, but it doesn't have to be.  One would think that if a CEO had created or overseen a project where the engineers (or marketing people...) pt money over lives... more than once, that CEO/management/engineers would be arrested and tried for hundreds of counts of murder.  The same goes for felony charges with investment and other fraud.  But, time and time again these laws are not enforced.  The CEO of Boeing still has his job and is still making millions.  All that does is encourage the behavior.  The real root cause is a complete lack of accountability, and these events will happen until that changes.  Judging by the long history of similar events, I'm not going to hold my breath.