Author Topic: don't forget to reboot your boeing, or else  (Read 1676 times)

0 Members and 1 Guest are viewing this topic.

Offline coppercone2

  • Super Contributor
  • ***
  • Posts: 4034
  • Country: us
  • $
don't forget to reboot your boeing, or else
« on: April 02, 2020, 11:50:28 pm »
https://tech.slashdot.org/story/20/04/02/206230/boeing-787s-must-be-turned-off-and-on-every-51-days-to-prevent-misleading-data-being-shown-to-pilots

Similar run time to windows ME?

its designed like the stations in the series Lost. Don't forget to press the button or else......
« Last Edit: April 03, 2020, 12:03:48 am by coppercone2 »
 

Offline Halcyon

  • Global Moderator
  • *****
  • Posts: 4016
  • Country: au
Re: don't forget to reboot your boeing, or else
« Reply #1 on: April 03, 2020, 01:12:30 am »
Reminds me more of Windows 10. Every week or so, I have to reboot my workstations at work because randomly, the USB ports stop working.
 

Offline Ampera

  • Super Contributor
  • ***
  • Posts: 2567
  • Country: us
    • Ampera's Forums
Re: don't forget to reboot your boeing, or else
« Reply #2 on: April 03, 2020, 02:32:52 am »
ha ha yes kick the windows man in the gutter ha ha
Professional complainer-in-chief criticizing other people's code
Programmer and bumbling Unix fool
Op @ EEVBlog IRC: irc.austnet.irc #eevblog
 

Offline floobydust

  • Super Contributor
  • ***
  • Posts: 3593
  • Country: ca
Re: don't forget to reboot your boeing, or else
« Reply #3 on: April 03, 2020, 02:36:58 am »
Is it the usual timer overflow somewhere, or maybe a memory leak? It doesn't happen during flight, right?

Boeing outsourcing their flight control software saved so many important dollars. It also gives many new grads entry to aerospace, provides backdoor cash flow to important customers in developing countries. The MBA strategic management strategy is pure genius.
 

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 6512
  • Country: gb
Re: don't forget to reboot your boeing, or else
« Reply #4 on: April 03, 2020, 02:41:50 am »
Is it the usual timer overflow somewhere, or maybe a memory leak? It doesn't happen during flight, right?

You're not generally interested in airspeed, attitude, or altitude when not in flight, so.. yes, it happens in flight.

Boeing, ladies and gentlemen..
 

Offline JustMeHere

  • Frequent Contributor
  • **
  • Posts: 289
  • Country: us
Re: don't forget to reboot your boeing, or else
« Reply #5 on: April 03, 2020, 03:31:10 am »
I was on an Airbus not too long ago.  The crew was late.   The co-pilot got there first, boarded and turned on the airplane so the air would circulate.  It took most of an hour for the rest of the crew to get there.  They went to push back and stopped.

Pilot: We are sorry for the delay you already have been through, but we have a warning light and will have to be here longer. 
Pilot: Well the manual says to reboot the airplane.  This will take about 5 minuets.
 

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 6512
  • Country: gb
Re: don't forget to reboot your boeing, or else
« Reply #6 on: April 03, 2020, 03:32:18 am »
I was on an Airbus not too long ago.  The crew was late.   The co-pilot got there first, boarded and turned on the airplane so the air would circulate.  It took most of an hour for the rest of the crew to get there.  They went to push back and stopped.

Pilot: We are sorry for the delay you already have been through, but we have a warning light and will have to be here longer. 
Pilot: Well the manual says to reboot the airplane.  This will take about 5 minuets.

Well at least the Airbus tells you when it needs to reboot instead of trying to fly you into a mountain!
 

Offline JustMeHere

  • Frequent Contributor
  • **
  • Posts: 289
  • Country: us
Re: don't forget to reboot your boeing, or else
« Reply #7 on: April 03, 2020, 03:36:19 am »
Is it the usual timer overflow somewhere, or maybe a memory leak? It doesn't happen during flight, right?

You're not generally interested in airspeed, attitude, or altitude when not in flight, so.. yes, it happens in flight.

Boeing, ladies and gentlemen..

Don't worry, the do turn the airplanes off at night.
 

Offline blueskull

  • Supporter
  • ****
  • Posts: 13880
  • Country: cn
  • Power Electronics Guy
Re: don't forget to reboot your boeing, or else
« Reply #8 on: April 03, 2020, 03:41:47 am »
2^32/24/3600/1000=49.7. Sounds like a 1ms tick counter overflow.
 
The following users thanked this post: Psi, GeorgeOfTheJungle, Jacon, syau, Alti

Offline floobydust

  • Super Contributor
  • ***
  • Posts: 3593
  • Country: ca
Re: don't forget to reboot your boeing, or else
« Reply #9 on: April 03, 2020, 05:43:37 am »
"The FAA has received a report indicating that the stale-data monitoring function of CCS {common core system (CCS) – a Wind River VxWorks realtime OS product} may be lost when continuously powered on for 51 days. This could lead to undetected or unannunciated loss of CDN message age validation, combined with a CDN switch failure. The CDN handles all the flight-critical data (including airspeed, altitude, attitude, and engine operation), and several potentially catastrophic failure scenarios can result from this situation. Potential consequences include:
• Display of misleading primary attitude data for both pilots.
• Display of misleading altitude on both pilots' primary flight displays (PFDs).
• Display of misleading airspeed data on both pilots' PFDs, without annunciation of failure, coupled with the loss of stall warning, or over-speed warning.
• Display of misleading engine operating indications on both engines.

The potential loss of the stale-data monitoring function of the CCS when continuously powered on for 51 days, if not addressed, could result in erroneous flight-critical data being routed and displayed as valid data, which could reduce the ability of the flightcrew to maintain the safe flight and landing of the airplane. " https://ad.easa.europa.eu/ad/US-2020-06-14

"Persistent or unfiltered stale data is a known 787 problem. In 2014 a Japan Airlines 787 caught fire because of the (entirely separate, and since fixed) lithium-ion battery problem. Investigators realised the black boxes had been recording false information, hampering their task, because they were falsely accepting stale old data as up-to-the-second real inputs."

"... another 787 stale data problem in years gone by saw superseded backup flight plans persisting in standby navigation computers, and activating occasionally. Activation caused the autopilot to wrongly decide it was halfway through flying a previous journey – and manoeuvre to regain the "correct" flight path." https://www.theregister.co.uk/2020/04/02/boeing_787_power_cycle_51_days_stale_data/

No wonder Boeing took a charge of $410M for a re-do test of the Starliner.
"... Mr. Loverro said that the problems were not a consequence of the fixed-price structure of the SpaceX and Boeing contracts. "I think it was the way we chose to manage it," Mr. Loverro said." :palm:
 
The following users thanked this post: tom66, I wanted a rude username

Offline Electro Detective

  • Super Contributor
  • ***
  • Posts: 2713
  • Country: au
Re: don't forget to reboot your boeing, or else
« Reply #10 on: April 03, 2020, 07:59:51 am »

or they could just swap out the BIOS backup battery every six months
instead of waiting for the Black Box to FYI the investigators  :popcorn:

Too easy, right?  :palm:
 

Online MK14

  • Super Contributor
  • ***
  • Posts: 2582
  • Country: gb
Re: don't forget to reboot your boeing, or else
« Reply #11 on: April 03, 2020, 08:17:35 am »
2^32/24/3600/1000=49.7. Sounds like a 1ms tick counter overflow.

I heard that, from another source. But the rumour goes further.
It uses the 1024/1000 (because some things do it that way, when converting) concept.

So 49.7 x 1024/1000 = 50.9 = A lot closer to the 51 days.

I.e. The 1 millisecond, may be a value (such as 1.024 ms per tick), so that rapid bit shifts can be used in calculations, rather than much slower full divides. Or some similar reason, why it is 1024/1000.

N.B. I didn't write the software for them. So can only guess at why it is 1024/1000. Which it could easily NOT be, anyway.

Anyway, I think it is completely crazy that they have to do that. So, if they miscount the 51 days (e.g. leap year), does the plane crash into a mountain and kill everyone on board ?

Why can't they fix the software, properly ?
(Yes, it would be expensive, but it seems a bit risky to leave a software bug like that, in the airplane).

Such problems, shouldn't of been designed into the plane in the first place and also, should have been found, in their careful tests and analysis of the safety critical software, on the planes systems.
This doesn't give me confidence, in their software.
 

Offline Yansi

  • Super Contributor
  • ***
  • Posts: 3290
  • Country: 00
  • STM32, STM8, AVR, 8051
Re: don't forget to reboot your boeing, or else
« Reply #12 on: April 03, 2020, 08:22:31 am »
Reminds me more of Windows 10. Every week or so, I have to reboot my workstations at work because randomly, the USB ports stop working.

Wau! And I thought it was only my computer that does it! Insane crap...
 

Offline Electro Detective

  • Super Contributor
  • ***
  • Posts: 2713
  • Country: au
Re: don't forget to reboot your boeing, or else
« Reply #13 on: April 03, 2020, 09:21:01 am »

"Windows has deactivated the USB sockets

to protect your computer due to user failure to update security on a regular basis.

If you want functionality restored and not see this message again,

YOU (expletive) KNOW WHAT TO (expletive) DO !

or visit    www.obeymsknowsbest.com

Have a nice day :) "


 
 

Offline senso

  • Frequent Contributor
  • **
  • Posts: 951
  • Country: pt
    • My AVR tutorials
Re: don't forget to reboot your boeing, or else
« Reply #14 on: April 03, 2020, 09:31:02 am »
2^32/24/3600/1000=49.7. Sounds like a 1ms tick counter overflow.

I heard that, from another source. But the rumour goes further.
It uses the 1024/1000 (because some things do it that way, when converting) concept.

So 49.7 x 1024/1000 = 50.9 = A lot closer to the 51 days.

I.e. The 1 millisecond, may be a value (such as 1.024 ms per tick), so that rapid bit shifts can be used in calculations, rather than much slower full divides. Or some similar reason, why it is 1024/1000.

N.B. I didn't write the software for them. So can only guess at why it is 1024/1000. Which it could easily NOT be, anyway.

Anyway, I think it is completely crazy that they have to do that. So, if they miscount the 51 days (e.g. leap year), does the plane crash into a mountain and kill everyone on board ?

Why can't they fix the software, properly ?
(Yes, it would be expensive, but it seems a bit risky to leave a software bug like that, in the airplane).

Such problems, shouldn't of been designed into the plane in the first place and also, should have been found, in their careful tests and analysis of the safety critical software, on the planes systems.
This doesn't give me confidence, in their software.

Due to certification, they would have to re-test all the software if its in fact a tick timer that will be used almost everywhere in the code..
And that is expensive, and the CxO can't go home without their 10+ million performance bonus..
 
The following users thanked this post: MK14

Online MK14

  • Super Contributor
  • ***
  • Posts: 2582
  • Country: gb
Re: don't forget to reboot your boeing, or else
« Reply #15 on: April 03, 2020, 09:50:09 am »
Due to certification, they would have to re-test all the software if its in fact a tick timer that will be used almost everywhere in the code..
And that is expensive, and the CxO can't go home without their 10+ million performance bonus..

But the air standards authorities, should INSIST, that it is done properly.

If there was a component, in the planes engine, which wore out every 51 days. The engine would explode and crash the plane, if they forgot to replace it.

I suspect, they (air standards authorities), would insist that the planes are recalled, and the "faulty" short life parts are redesigned, to last the proper/expected life expectancies (e.g. life of the engines).

Software, should be the same.

E.g. The planes will likely be in storage or unused, for LONGER than 51 days, because of the virus. So, if it had been left on (e.g. some planes kept on, in standby (or whatever it is called, in plane jargon), for various reasons, so it can be ready quickly. I heard this a long time ago. It usually applies to military planes, especially), and needed to be rebooted. It could just take one plane, where they forget to reboot (or similar mishap), and it could spell disaster.
In this case (extended coronavirus long term shutdown, many weeks or months), the aircraft would probably be shutdown/off.
« Last Edit: April 03, 2020, 09:53:36 am by MK14 »
 

Offline SeanB

  • Super Contributor
  • ***
  • Posts: 15402
  • Country: za
Re: don't forget to reboot your boeing, or else
« Reply #16 on: April 03, 2020, 09:52:29 am »
A lot of the flight subsystems are rerely turned off, and are running all the time between services, and often also during service pperiods, as the plane will be powered from an external power unit then.  Reboot of some systems can be fast, because it is a warm reboot, but powering down, and doing a full restart can take hours, as you have to bring things back up in sequence, so you do things like first connect the main battery, then turn on the GPS and wait for it to acquire lock, so your main data bus systems will have a stable clock and accurate time data. Then you can start powering up the systems in sequence, such that each one communicates with the units it depends on to get valid inputs. finally you can power up the FMS and displays, which is the top layer the pilots see.

That at least reboots fast, but only if the underlying systems are running, without having exceeded the abilities of the redundant subsystems. Things will fail all the time, and there are a whole lot of fault trees that you follow, to see if you can continue to destination without degradation, continue with degradation, divert to alternate airport soon, land as soon as possible, land immediately or " kiss your backside goodbye, because that is it folks", depending on the fault and just how many systems it affects. Then you get mechanical failures, with the same fault trees, but at least you can hope the electronics is going to give you a true indication of the system status.

51 days of uptime then it breaks is a symptom of poor testing, just like the engine management turning off the alternators after 250 days of uptime. Both of which are common to have, as those systems run all the time, even with the plane on the ground in a standby mode. Likely to have been introduced as a consequence of another software bug being fixed some time ago, or to be a result of errors in interpretation of inputs, or validation of parameters, or just insufficient testing of all possible program paths.

You might turn off the lighting on the ground, and also turn off the APU, and have a cockpit that is totally black, but there are still computer systems running to enable you to turn on a switch, and have the lights come on, and it to warm start in the morning, but the uptime on the computers will be that since the last time the battery was disconnected along with the external power connections.

Only good thing is all the grounded 737MAX aircraft will have been parked with the main battery breakers turned off in the avionics bays, so as not to overdischarge the batteries, or they were removed so they could be put on maintenance charge schedule. They will be fine for the first 51 days after they fly again, at least for this bug, and perhaps even MCAS issues will be properly fixed as well.

I will add that military aircraft rarely are powered for long times, as they have in general a much more frequent service schedule, due to parts being run much closer to the limit as compared to commercial aircraft, where they want parts to do long periods between services, and even longer between overhauls. Thus your jet engine might run 90 days between service intervals, or 500 running hours, while your military version of the same engine will be monthly, or 200 hours, due to the higher demands placed on it.  Even in hot standby the same aircraft will rarely be run for more than a week continuously, before another is swapped in to replace it, while the first goes for service. In a situation of active military use though they might just be run till they break, but that is where you find that they are also going to be spares part sources quickly, to keep others running.

Longest I saw was 2 weeks of continuous operation, in use 8 hours a day, with hot swapping pilots, because they were down to only having one aircraft available, the others being not available for various reasons, and the pilots needed the hours to keep flying. It did not break, though the engine came in right afterwards, well past the service hours count. I had no faults off it either, the avoinics i was responsible for worked correctly.
« Last Edit: April 03, 2020, 10:02:26 am by SeanB »
 
The following users thanked this post: MK14

Offline I wanted a rude username

  • Frequent Contributor
  • **
  • Posts: 393
  • Country: au
  • ... but this username is also acceptable.
Re: don't forget to reboot your boeing, or else
« Reply #17 on: April 03, 2020, 08:46:48 pm »
It uses the 1024/1000 (because some things do it that way, when converting) concept.

I bet it just counts microseconds:

  • 2^42 microseconds lines up with the time reported
  • Maybe a 10-bit sub-second microsecond counter and a 32-bit second millisecond counter
    • The sub-second counter could be shifted and masked from a hardware cycle counter
    • Less likely, the system adds one tick's worth of microseconds each tick

Like OP, I also immediately thought of the old Windows 49.7 day crash.

Edit: In my defence, it made sense in my head I am legally retarded.
« Last Edit: April 04, 2020, 03:22:26 am by I wanted a rude username »
 

Online MK14

  • Super Contributor
  • ***
  • Posts: 2582
  • Country: gb
Re: don't forget to reboot your boeing, or else
« Reply #18 on: April 04, 2020, 12:29:05 am »
I bet it just counts microseconds:

  • 2^42 microseconds lines up with the time reported
  • It probably has a 10-bit sub-second counter and a 32-bit second counter
  • The sub-second counter is likely shifted and masked from a hardware cycle counter
    • Less likely, the system adds one tick's worth of microseconds each tick

Your maths, doesn't seem to add up.
Because a 10 bit divider, would be milliseconds, approximately, NOT microseconds. 10 bit = 1024.

a 32-bit second counter

I'm NOT quite understanding your maths, here.
2^32 seconds = Over 136 Years!

But, anyway. Further research seems to indicate that the software was actually fixed, later (maybe the 51 days, was ONLY until the software was fixed and recertified).

The OS they seem to be reportedly using, VxWorks, seems to use 50 or 60Hz, as the Timer Tick.
= 16.666ms (60Hz) approx. But it does seem to mention something about hardware timer resolution can be different.

So, I'm not really sure how the 51 days comes about.
« Last Edit: April 04, 2020, 12:43:55 am by MK14 »
 

Offline coppercone2

  • Super Contributor
  • ***
  • Posts: 4034
  • Country: us
  • $
Re: don't forget to reboot your boeing, or else
« Reply #19 on: April 04, 2020, 02:53:57 am »
it just could be a sum of data samples that fills up a memory space from multiple sensors or something (does not make sense to sample some sensors at ultra high frequencies probably). Maybe it makes a SOA from different sensors data inputs.
 

Offline MathWizard

  • Regular Contributor
  • *
  • Posts: 120
  • Country: ca
Re: don't forget to reboot your boeing, or else
« Reply #20 on: April 04, 2020, 03:06:12 am »
Boeing, what an enigma. They make bombs to kill 3rd world people, to help make rich people richer, and they make nice planes, overall.
 

Online MK14

  • Super Contributor
  • ***
  • Posts: 2582
  • Country: gb
Re: don't forget to reboot your boeing, or else
« Reply #21 on: April 04, 2020, 03:11:31 am »
it just could be a sum of data samples that fills up a memory space from multiple sensors or something (does not make sense to sample some sensors at ultra high frequencies probably). Maybe it makes a SOA from different sensors data inputs.

(To avoid confusion, I'm NOT attacking your post. Just describing the difficulty of trying to determine someone else's mistake/bug, when the full information is apparently not available).
Yes, we could speculate, on the millions of ways, they could have messed up. But, it is like finding a needle in a haystack.

Although I've not seen the precise details. There is some details on the specifics of the bug, floating around.

Quote
the 787's common core system (CCS) stops filtering out stale data from key flight control displays. That stale data-monitoring function going down in turn "could lead to undetected or unannunciated loss of common data network (CDN) message age validation, combined with a CDN switch failure".

https://www.theregister.co.uk/2020/04/02/boeing_787_power_cycle_51_days_stale_data/
« Last Edit: April 04, 2020, 03:33:49 am by MK14 »
 

Offline duak

  • Super Contributor
  • ***
  • Posts: 1009
  • Country: ca
Re: don't forget to reboot your boeing, or else
« Reply #22 on: April 04, 2020, 07:23:38 pm »
One company I worked at had a number of aviation buffs.  One of them said something interesting about passenger aircraft - I don't know if it's factually true or just truthy.  He said that McDonnell-Douglas had two hydraulic systems, Lockheed had three and Boeing had four.  This happened at a time when M-D had a number of spectacular failures in the DC-10.  BTW, he worked for McDonnell before they were merged with Douglas.  This probably contributed to me being more confident in Boeing.

I've flown about 75 times domestically and overseas over the years.  The most exciting mechanical failure was a hydraulic system crapout in a Fokker.  The crew had to crank down the gear, and we circled until the emergency team got in place.  It took a bit longer to come to a stop but if you didn't know something was up, you'd wonder what they were worried about.  It was much more exciting to land in Jakarta, Indonesia during a thunderstorm in a DC-10 or the dogleg final approach at Hong Kong's old Kai Tak airport in a 747.  A crosswind landing at SFO will also wake you up.  I don't recall having been in a missed approach.

In 1997, my wife and I flew from Seattle to LAX in a newish Alaskan Airlines 737.  We were cleared to go, the engines spooled up then just spooled down.  No bangs or funny noises.  A few seconds later the engines spooled up then just spooled down again.  Then a long pregnant pause before the captain came on and said we'd head back to the gate.  About an hour later, the captain came on and said the tech had reset the computer a few times but still had a problem so the computer was changed out.   I remember something a colleague said when asked why he didn't like to fly.  "Planes have computers with firmware in them".  BTW, both he and I designed hardware and wrote the firmware.  I suppose I was thinking about the fallibility of computers and whether this was a warning to get off the aircraft.  The flight was uneventful, but I wouldn't have been surprised if the engines decided to spool down and we had to glide in somewhere - assuming nothing else developed.  Fly by wire indeed!
« Last Edit: April 04, 2020, 08:10:14 pm by duak »
 
The following users thanked this post: MK14, I wanted a rude username

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 5453
  • Country: fr
Re: don't forget to reboot your boeing, or else
« Reply #23 on: April 05, 2020, 01:05:42 am »
Whatever the technical reason at the implementation level, what is concerning IMO is that it never got caught during any test; either that, or it was actually specified this way, or not specified at all.

So we can debate/guess internal counter widths or buffer sizes to no end, but the specification, verification and/or validation stage clearly failed somewhere.
 

Offline JustMeHere

  • Frequent Contributor
  • **
  • Posts: 289
  • Country: us
Re: don't forget to reboot your boeing, or else
« Reply #24 on: April 06, 2020, 07:04:14 pm »
Boeing, what an enigma. They make bombs to kill 3rd world people, to help make rich people richer, and they make nice planes, overall.

Boeing has never made a bomb.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf