Author Topic: Another deadly 737 Max control bug just found!  (Read 11762 times)

0 Members and 1 Guest are viewing this topic.

Online Mr. Scram

  • Super Contributor
  • ***
  • Posts: 9337
  • Country: 00
  • Display aficionado
Re: Another deadly 737 Max control bug just found!
« Reply #75 on: July 10, 2019, 03:54:12 pm »
And why machines for which the state cannot be documented due to things like heap allocation should not be used in safety critical applications.  This also makes processor features which contain unknown state like caches, speculative execution, and multi-threading less desirable.
That ship has sailed. Literally. They use Windows to run warships.

https://www.theregister.co.uk/2007/02/26/windows_boxes_at_sea/
« Last Edit: July 10, 2019, 04:00:07 pm by Mr. Scram »
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 11250
  • Country: us
  • DavidH
Re: Another deadly 737 Max control bug just found!
« Reply #76 on: July 11, 2019, 03:14:10 am »
And why machines for which the state cannot be documented due to things like heap allocation should not be used in safety critical applications.  This also makes processor features which contain unknown state like caches, speculative execution, and multi-threading less desirable.

That ship has sailed. Literally. They use Windows to run warships.

https://www.theregister.co.uk/2007/02/26/windows_boxes_at_sea/

Back in the mid 1980s, a Ticonderoga class cruiser off of San Diego lost all power to this problem.  The only illumination they had was flashlights and they had to be towed back to port.

 

Offline windsmurf

  • Frequent Contributor
  • **
  • !
  • Posts: 626
  • Country: us
Re: Another deadly 737 Max control bug just found!
« Reply #77 on: July 15, 2019, 01:31:54 am »
American Airlines and United Airlines extend 737 Max grounding through at least early November

https://www.latimes.com/business/la-fi-boeing-max-american-united-grounding-20190714-story.html

 

Offline floobydust

  • Super Contributor
  • ***
  • Posts: 3530
  • Country: ca
Re: Another deadly 737 Max control bug just found!
« Reply #78 on: July 15, 2019, 03:36:04 am »
That's 8,000 flights for United Airlines $185M revenue. I say the 737 max won't be flying until well into 2020.
I wonder if the money and lives lost will ever be a motivator to do it right, or will it be just investor's cash being burned for a little "hiccup".

Instead of parking lots full, maybe Boeing could turn the planes into condos? Just park under the wings.
 

Offline aix

  • Regular Contributor
  • *
  • Posts: 136
  • Country: gb
Re: Another deadly 737 Max control bug just found!
« Reply #79 on: July 16, 2019, 07:31:58 am »
Looks like there's a rebranding exercise going on: https://www.bbc.co.uk/news/business-48995509
 

Offline Gyro

  • Super Contributor
  • ***
  • Posts: 5626
  • Country: gb
Re: Another deadly 737 Max control bug just found!
« Reply #80 on: July 16, 2019, 08:49:19 am »
Not sure if it's a knock-on but Ryanair are now saying that they will have to cut flights and possibly bases next year as a result of the delay...

https://www.bbc.co.uk/news/business-49000796

It would be amazing if they were actually trying to get away with rebranding the Max. Ryanair have a pretty poor reputation.
Chris

"Victor Meldrew, the Crimson Avenger!"
 

Offline splin

  • Frequent Contributor
  • **
  • Posts: 995
  • Country: gb
Re: Another deadly 737 Max control bug just found!
« Reply #81 on: July 17, 2019, 01:23:23 am »
And why machines for which the state cannot be documented due to things like heap allocation should not be used in safety critical applications.  This also makes processor features which contain unknown state like caches, speculative execution, and multi-threading less desirable.

Except that pretty much any non-trivial system will use heap allocation - but it will be called something else, typically a buffer pool or the like. Try, for example, implementing a comms protocol without one. These pools will be safer than a global heap because (at least):

a) They typically will use fixed size allocations (or a limited number of fixed sizes from different pools) so should be free from heap fragmentation which is one of the bigger problems with heaps.

b) They will also be shared between a limited subset of the whole application.

Typically this managed memory will be passed between tasks/processes/subsystems - eg. between different layers in a comms stack, including device drivers, with all the attendant risks due to the distance and time seperating the allocator of the memory, the users and the consumer that has to de-allocate the memory.

By distance, I mean between developers, who may be in different teams, requiring coordination/documentation and suitable development tools to try to ensure that problems such as memory leaks, accessing free'd off memory etc. are minimized (I'd say eliminated if that were possible).

You can improve things by copying buffers between tasks/subsystems if you can afford the performance and memory cost - but how many systems have the luxury? If you have finite resources someone has to decide how to allocate it. More stack or more buffer space? In the case of Toyota, they didn't allocate enough for stack space but had they increased the stack allocation something else would have less memory available - ok if you can determine in advance the worst case memory usage but there are many times when static analysis tools can't be used and it comes down to the skill of the developers to work out the worst cases.

For safety critical applications you obviously must be able to guarantee the behaviour of the critical parts of the system and isolate them from less trusted subsystems. The reality is that tradeoffs between cost, development time/effort (re-certifying a 737 MAX), functionality and safety are always being made - there rarely absolutes - if ever. No point in a seven-nines reliability requirement if it costs ten trillion dollars. Or a car ECU which costs $1500 and can't do MP3. An aeroplane with enough redundancy to be guaranteed never to fail is probably too heavy to fly. Software that takes longer to develop than the life cycle of the product it's used on is pointless.
 

Online Rick Law

  • Super Contributor
  • ***
  • Posts: 2834
  • Country: us
Re: Another deadly 737 Max control bug just found!
« Reply #82 on: July 17, 2019, 02:40:10 am »
...
...
 Software that takes longer to develop than the life cycle of the product it's used on is pointless.

I think software problem is more because of it being "lower entry barrier."  Anyone with enough money to buy a cheap laptop can start developing software.  Compare the cost of starting a new software development company to say cost of starting a company to develop a new video chip to compete with NVIDIA.  The ratio of the two cost-of-entry would (with exaggeration) overflow a typical calculator.

This low entry cost of software development leads to the average quality of the developer, quality of the development shop, experience of the development shop all correspondingly lowered.  You could, in theory, have people writing software for a washing machine who has never seen a working washing machine, and they are writing their washing machine software in a shed next to a river bank next to the bunch of people washing their cloths on the river shore.  You could do software in that environment, but you are not going to develop any new IC's of complexity sitting in shed with bathrooms in the out-house.

Yeah, I exaggerated.  I exaggerate to point out how the lower entry barrier can bring in much lower caliber teams.

...
I wonder if the money and lives lost will ever be a motivator to do it right, or will it be just investor's cash being burned for a little "hiccup".

Instead of parking lots full, maybe Boeing could turn the planes into condos? Just park under the wings.

They will remember it for a while, sack some of the ones involved (who merely followed order too do everything needed to reduce cost), and bring in new blood.  Hiring these new blood will be announced as investment to make sure such problem will be fixed and make sure such fiasco will never happen again.  As their stock improves (even if the stock price increase is due to improvement in the general economy or due to inflation), their memory fade as the stock price rises...

The incoming new blood will of course hire more new guys.  Most important of the new hires will be the ones for the new marketing team - the team that will do magic to wash off the stain of "sins of prior management team" in the public's (and the customer's) eyes.  But, the new marketing drive needs funding.  So, budget cut for the development groups to fund the new marketing drives.  They did a bad job that caused the problem, so they deserve to have their budget cut to the bone and then some.  They will be told to find ways to do software development "smarter".  That is to say, do it faster, better, and with even less money.  If the development team(s) can't do that, we fire the whole lot and outsource the whole darn thing.
 

Online GeorgeOfTheJungle

  • Super Contributor
  • ***
  • Posts: 2493
  • Country: tr
Re: Another deadly 737 Max control bug just found!
« Reply #83 on: July 17, 2019, 07:27:00 am »
You could do software in that environment, but you are not going to develop any new IC's of complexity sitting in shed with bathrooms in the out-house.

Look: youtube.com/watch?v=QqxThgLTLyk&t=7m54s :-)

Part one: youtube.com/watch?v=jhwwrSaHdh8 and part two: youtube.com/watch?v=re5xAqgKqc0
http://brave.com <- BETTER AND FASTER BROWSER. YOUTUBE W/O ADS/INTERRUPTIONS.
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 11250
  • Country: us
  • DavidH
Re: Another deadly 737 Max control bug just found!
« Reply #84 on: July 18, 2019, 03:23:43 am »
And why machines for which the state cannot be documented due to things like heap allocation should not be used in safety critical applications.  This also makes processor features which contain unknown state like caches, speculative execution, and multi-threading less desirable.

Except that pretty much any non-trivial system will use heap allocation - but it will be called something else, typically a buffer pool or the like. Try, for example, implementing a comms protocol without one. These pools will be safer than a global heap because (at least):

...

How complex does a system involving physical control loops need to be?  Separate it and stick it on redundant hardware.

Static allocations are wasteful but so what?  Memory is cheap.

Quote
For safety critical applications you obviously must be able to guarantee the behaviour of the critical parts of the system and isolate them from less trusted subsystems. The reality is that tradeoffs between cost, development time/effort (re-certifying a 737 MAX), functionality and safety are always being made - there rarely absolutes - if ever. No point in a seven-nines reliability requirement if it costs ten trillion dollars. Or a car ECU which costs $1500 and can't do MP3. An aeroplane with enough redundancy to be guaranteed never to fail is probably too heavy to fly. Software that takes longer to develop than the life cycle of the product it's used on is pointless.

I suspect the larger problem is that the programming techniques are no longer taught and the tools are no longer available.

In this case none of that mattered because the system operated exactly as designed.  It was just designed ineptly.
 
The following users thanked this post: KL27x

Online coppercone2

  • Super Contributor
  • ***
  • Posts: 3925
  • Country: us
  • $
Re: Another deadly 737 Max control bug just found!
« Reply #85 on: July 25, 2019, 10:39:02 pm »
there is a obsession with memory space in conventional software design that should not be carried over to embedded systems. Even with code clarity.
 

Offline windsmurf

  • Frequent Contributor
  • **
  • !
  • Posts: 626
  • Country: us
Re: Another deadly 737 Max control bug just found!
« Reply #86 on: July 25, 2019, 11:03:50 pm »

Speaking of memory space, memory overflow is forcing Airbus planes to power cycle the entire avionics suite every 149 hours

https://www.theregister.co.uk/2019/07/25/a350_power_cycle_software_bug_149_hours/
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 11250
  • Country: us
  • DavidH
Re: Another deadly 737 Max control bug just found!
« Reply #87 on: July 25, 2019, 11:53:33 pm »
That reminds me of the accumulation of residual errors causing problems with the Patriot missile system:

https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dhahran
 

Offline BravoV

  • Super Contributor
  • ***
  • Posts: 6801
  • Country: 00
  • +++ ATH1
Re: Another deadly 737 Max control bug just found!
« Reply #88 on: July 26, 2019, 05:21:56 am »

Speaking of memory space, memory overflow is forcing Airbus planes to power cycle the entire avionics suite every 149 hours

https://www.theregister.co.uk/2019/07/25/a350_power_cycle_software_bug_149_hours/

I guess Windows 95 or 98 are much better as they only need to be restarted at every 49.7 days.  :-DD

Online GeorgeOfTheJungle

  • Super Contributor
  • ***
  • Posts: 2493
  • Country: tr
Re: Another deadly 737 Max control bug just found!
« Reply #89 on: July 26, 2019, 07:31:04 am »
(2^32)/1000/3600/24= 49.7
http://brave.com <- BETTER AND FASTER BROWSER. YOUTUBE W/O ADS/INTERRUPTIONS.
 

Offline Kleinstein

  • Super Contributor
  • ***
  • Posts: 7555
  • Country: de
Re: Another deadly 737 Max control bug just found!
« Reply #90 on: July 26, 2019, 08:57:51 am »
there is a obsession with memory space in conventional software design that should not be carried over to embedded systems. Even with code clarity.
With modern PC software they don't seem to care abut memory anymore. Software gets bloated so much it no longer runs on older system and update take forever with a slow internet connection. Also the feeling is that software quality tends to go down the larger a program - kind of normal with the time needed for debugging going up usually more like the 3rd power or even exponential with code length.

Software development can be quite different for a modern multi tasking OS, an old PC and embedded systems with OS (e.g. RTOS) and embedded without an OS. Especially for a critical systems things like static memory allocations and maybe not using pointers at all can be a good solution. Dynamic memory can cause odd effects, like fragmented memory or a garbage collector to sometimes take a long time.

Those programmers that started with core rope, mask-ROMs and one time programmable µCs now are retiring.
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 11250
  • Country: us
  • DavidH
Re: Another deadly 737 Max control bug just found!
« Reply #91 on: July 26, 2019, 01:38:01 pm »
We found an bug in the IP stack of Windows 2000 which slowly leaked memory requiring a reboot about every 2 weeks on a low memory system.  Microsoft's only recommendation was to install more memory and they never fixed it.
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 5345
  • Country: fr
Re: Another deadly 737 Max control bug just found!
« Reply #92 on: July 26, 2019, 01:51:41 pm »
We found an bug in the IP stack of Windows 2000 which slowly leaked memory requiring a reboot about every 2 weeks on a low memory system.  Microsoft's only recommendation was to install more memory and they never fixed it.

Is that what we call being "practical"? ;D
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 5345
  • Country: fr
Re: Another deadly 737 Max control bug just found!
« Reply #93 on: July 26, 2019, 02:02:27 pm »
How complex does a system involving physical control loops need to be?  Separate it and stick it on redundant hardware.

Static allocations are wasteful but so what?  Memory is cheap.

Yeah I agree. Control loops in avionics are still based on pretty simple algorithms. No need for any dynamic allocation.

MISRA-C is clear about that.
https://www.misra.org.uk/forum/viewtopic.php?t=1233

I believe the rules are, or at least should reasonably be the same in avionics software.

If using dynamic allocation, you would at least have to justify that thoroughly. So I'm wondering what kind of arguments could be used here.
I know some teams heavily rely on automated tools (such as dynamic code analysis), so that if the tools don't catch any potential memory leak, the team considers the code OK. That gives them a feeling of a safety net and gives them the possibility of using dynamic allocation without having to think twice about it (thinking the tools will catch all problems). This is bad.

On a general level, following no or fewer rules just because there are machines or people that are going to check an engineer's work is a recipe for bad design...
 
The following users thanked this post: KL27x, Siwastaja

Offline StillTrying

  • Super Contributor
  • ***
  • Posts: 2557
  • Country: dk
  • Country: Broken Britain
Re: Another deadly 737 Max control bug just found!
« Reply #94 on: July 28, 2019, 06:43:51 pm »
UK TV tomorrow night.

Boeing's Killer Planes
Panorama
The Boeing 737 Max was set to dominate the skies, but two deadly crashes have left the aircraft grounded and the company in crisis. Both flights were forced down by software specially designed for the new aircraft. Reporter Richard Bilton investigates the plane’s fatal flaws and asks whether Boeing should have done more to protect passengers.
https://www.bbc.co.uk/programmes/m00077cw
CML+  That took much longer than I thought it would.
 

Offline MT

  • Super Contributor
  • ***
  • Posts: 1290
  • Country: cn
Re: Another deadly 737 Max control bug just found!
« Reply #95 on: July 28, 2019, 07:45:18 pm »
How did that tittle even pass BBC program board executives? ;D  BBC seams desperate these days yet again connected to pedophilia and all!
 

Online Towger

  • Super Contributor
  • ***
  • Posts: 1577
  • Country: ie
Re: Another deadly 737 Max control bug just found!
« Reply #96 on: July 29, 2019, 06:57:09 am »
The BBC culture of old appears to have been left behind in London when it sold up and moved to Manchester.

I am sure Trump will have an interesting tweet about the program...
 

Offline CiscERsang

  • Regular Contributor
  • *
  • Posts: 119
  • Country: za
Re: Another deadly 737 Max control bug just found!
« Reply #97 on: July 29, 2019, 07:05:59 am »
The BBC culture of old appears to have been left behind in London when it sold up and moved to Manchester.

I am sure Trump will have an interesting tweet about the program...

His team is already looking for variants on helping to Boeing corporation to overcome the crisis, I believe.
 

Online Towger

  • Super Contributor
  • ***
  • Posts: 1577
  • Country: ie
Re: Another deadly 737 Max control bug just found!
« Reply #98 on: July 29, 2019, 09:02:27 am »

Quote
A former Boeing engineer has told the BBC's Panorama programme that work on the production line of the 737 Max plane was not adequately funded.

https://www.bbc.com/news/business-49142761
 

Offline CiscERsang

  • Regular Contributor
  • *
  • Posts: 119
  • Country: za
Re: Another deadly 737 Max control bug just found!
« Reply #99 on: July 29, 2019, 09:30:23 am »

Quote
A former Boeing engineer has told the BBC's Panorama programme that work on the production line of the 737 Max plane was not adequately funded.

https://www.bbc.com/news/business-49142761

 :palm: Shoked by details revealed. That's deadly side effect of capitalism.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf