Toyota firmware fail

[Harvs]

As a motorcyclist, a failure in a system like this makes hairs stand on end.  It's one of the reasons bikes have (generally) always had dual cables for the throttle, so it wouldn't just return under spring tension.

But as with cars, the trend is slowly to move over to drive by wire throttle systems.  This has allowed some even more stupidly high cam tuning, as it allows the ECU to make the bike (engine) somewhat drivable at low rpm.

However, imagine being on a new R1 et al and the throttle suddenly slams open uncommanded.  Yes you've got a clutch right by your left hand, but if you're in first or second you'll probably have the front wheel in the air and be at >200kph in a few sec time.  Chances of you hitting something hard before getting it under control are very good.

[David_AVD]

Speaking as someone who's had that "UA" experience, my first instinct was to go for the brake.

The very next thought was to turn the ignition off (to accessories position, not lock!), but then I recognised that the UA was linked to switching the A/C off, so switching it back on averted more drastic immediate action.

In hindsight I could have also popped it into neutral and let the rev limiter kick in while I braked to a stop.  Luckily there was nobody close in front of me at the time.  If it had happened in heavy traffic there might have been a small collision.

[walshms]

As a motorcyclist, a failure in a system like this makes hairs stand on end.

As a fellow motorcyclist, and former MSF Ridercoach, I can tell you that training is the only way to actually protect yourself.  Developing good habits, and experience, would save you in that case.  TCLOCS was something I stressed in classes, and understanding the machine thoroughly arms you for the unexpected.

You can downshift without a clutch, though it'll be a rough ride, and if you've got one of the more recent bikes, your brakes will function down to the point where they're nearly empty of fluid.  If you've done your pre-ride inspection properly, you're probably okay.

[walshms]

Oh, and of course, that handy-dandy engine cutoff switch right there by your right thumb. 

[SeanB]

Not helped by the "Oh we made this thing keyless so you do not have the inconvenience of needing to find the slot just press this button" who decided that you need to hold the button for 5 seconds to switch off, and also decided that if you press the button while in motion it will be disregarded as an accidental press.

[andtfoot]

Not helped by the "Oh we made this thing keyless so you do not have the inconvenience of needing to find the slot just press this button" who decided that you need to hold the button for 5 seconds to switch off, and also decided that if you press the button while in motion it will be disregarded as an accidental press.

I wonder what would happen if you threw the wireless key fob thingie out the window while driving...

[Stonent]

Not helped by the "Oh we made this thing keyless so you do not have the inconvenience of needing to find the slot just press this button" who decided that you need to hold the button for 5 seconds to switch off, and also decided that if you press the button while in motion it will be disregarded as an accidental press.

I wonder what would happen if you threw the wireless key fob thingie out the window while driving...

That would be interesting to try if you were on a street with no traffic, toss it out the window into the grass and see if it will keep going. I suspect it might because it would just waste the battery to keep in contact with it.

[Harvs]

That would be interesting to try if you were on a street with no traffic, toss it out the window into the grass and see if it will keep going. I suspect it might because it would just waste the battery to keep in contact with it.

Or if it was your car, you could just pass it out the window to a friend

[chickenHeadKnob]

 I can recall when the Prius runaway problem hit the news Steve Wozniak chimed in and claimed his prius was affected, Here is a youtube clip:
http://youtu.be/u44XjkWgFac    Read some of the comments, it seems Woz was confused by the cruise control.
 I partially remember reading there was a strong driver age bias (over 60 years old) to the cohort of un-commanded accelleration complainers.

In the early 80's when GM was a ghung-ho  early adopter of microprocessors  everywhere I heard about a software bug in one of the Cadillac models. When accelerating uphill or under load and transitioning through the middle gears - auto trans, if the driver then honked the horn the engine would suddenly lose power.

Diesel engines are reluctant to shut down once they are running, I think the standard design is to cut the fuel supply to the common fuel rail, but sometimes even that isn't sufficient. VW's first generation 4 cylinder engine installed in Rabbits in the mid 70's replicated a noob design flaw also exhibited by a few early american truck engines in 50's. They would, on rare occasion, cannilbalize  their own crankcase oil as fuel if the cylinders or rings were worn. When that happened the engine would race to 6000 rpm and seize in short order, but not before leaving the driver with an intense and bewildering experience, no software required.

[Stonent]

2 stroke Detroit diesel engines would eat their own oil sometimes. They used to make a kit you could install where if the suction on the intake got high enough due to overspeed or runaway, it would suck a trap door closed over the intake pipe and cut off all air.

[andersm]

Normally people here are ready to jump on anything and call it terrible, but now that there's evidence of bad design the board suddenly turns into the Toyota defense force?

[Stonent]

That would be interesting to try if you were on a street with no traffic, toss it out the window into the grass and see if it will keep going. I suspect it might because it would just waste the battery to keep in contact with it.

Or if it was your car, you could just pass it out the window to a friend

Doh! Much easier!

[dr.diesel]

Normally people here are ready to jump on anything and call it terrible, but now that there's evidence of bad design the board suddenly turns into the Toyota defense force?

Agreed.

While there is always two sides of every story, it's likely that Toyota dropped the ball on this one.  I imagine Toyota is similar to most other huge corporate slugs, cost cutting till it starts to hurt, or failing to invest where necessary.  Current gen ECMs are a crucial part of the vehicle and probably haven't received the time/money/development necessary.

[grumpydoc]

By the time memory exceptions came in we had MMUs anyway, which could do the same thing with a downward growing stack.

The difficulty here is that to handle the exception typically the first thing you need to do is extend the stack......

I have a problem with safety critical code and "exceptions" anyway - do you really want safety critical code to be saying "oops, didn't expect that"

[AlfBaz]

I have a problem with safety critical code and "exceptions" anyway - do you really want safety critical code to be saying "oops, didn't expect that"

Code: [Select]

if(oops_didnt_expect_that)
ShutDownEngine();
Surely better than having application data obliterated obliviously

[dr.diesel]

I have a problem with safety critical code and "exceptions" anyway - do you really want safety critical code to be saying "oops, didn't expect that"

Code: [Select]

if(oops_didnt_expect_that)
ShutDownEngine();
Surely better than having application data obliterated obliviously

What if this happens when your attempting to get off the train tracks, or get out of the way of a semi?  Eventually this condition will happen and the ECM must be prepared for it, exception or not.

[amyk]

I agree with those saying this is sensationalistic -- nowhere in the article does it say they actually found the cause, just some questionable marginal design (which I am almost willing to bet would be the same with various other manufacturers.)

Toyota claimed only 41% of the allocated stack space was being used. Barr's investigation showed that 94% was closer to the truth.

94% is still <= 100%. If they could prove that it never got over 100%, then that's not a point of failure to me.

My car's ECU only runs the fuel injection and the throttle is mechanical. I once drove a friend's recent "drive-by-wire" car and the sensation was a bit disturbing. At times the accelerator pedal felt like it was completely disconnected and the car had a mind of its own (which it does...)

[Dr. Frank]

The report in EDN is very strange in several  aspects:

1. Toyota is a car manufacturer, they do not (to my knowledge) design and manufacture car electronics.
The real supplier of the ECU is not named - strange.

2. The NASA had already analysed the software of this ECU, in a previous trial, and found no such bugs.
The driver was blamed of maloperation in first instance.

3. Such a relatively small and perhaps unexperienced  (in terms of automotive business or saftey design) SW engineer group should really find bugs, which the NASA team did not see??

4. All Tier 1 suppliers for OEMs have to follow strictly on design processes for Hardware and Software, which are standard in the automotive industry, e.g. according to CMMI and other models. A supplier, which does not conform to such requirements, would not work for OEMs, especially not for Toyota, which is a super-critical customer for electronic manufacturers.

Toyota normally audits those suppliers very strictly, especially on saftey electronics.

I can hardly believe, that such a safety component should not have been designed with extra effort,  and care in the usual validation tests.

And I also doubt, that such severe software design errors, or the lack of solid design rules and a lack of a sophisticated software system should be present in this case.


Frank

[0xdeadbeef]

Yeah, the EDN article is weird in many ways. Claiming that ECC RAM (never heard the term "EDAC" anywhere before) was state of the art in an ECU ("ECM") from 2005 (if I figure this right) seems a bit weird. E.g. Tricore controllers had only simple parity check until recently and some years ago even this had to be disabled for some models due to according errata.
I refuse to believe that no stack measurement was done, as this is so stupidly easy to do (initialize with stack pattern, check where from the top the first pattern was overwritten) that it was done in every project I ever saw in the last 15 years. And static stack analysis is kinda futile as it will always result in catastrophically high stack utilization. Besides, it's kinda hard to believe that a stack overflow resulted in acceleration instead of a reset.

Other stuff like the paradigm of "mirroring all important data" is questionable at least. Apart from the runtime hit when mirroring all important variables and consistency issues: what should you do if the main value and the mirrored value differ? Which one would you trust? Stuff like this was used for NVRAM or reset safe values in the past (instead of proper CRC), but just to detect inconsistency and return to a default value. For inconsistent runtime variables, you could only cause a reset from my point of view. Anyway, monitoring variables instead of monitoring functions seems like a bad idea. At least I'm not aware that this is part of any automotive safety concept. And probably for a good reason.

MISRA-C rule violations don't mean anything without further details. MISRA is very strict and a lot of rules are questionable at best. Anyway, a MISRA violation doesn't mean that the code is not working correctly. It can be just a "wrong" C++ comment, a missing (needless) cast, a goto (even if it's the best solution in some cases, e.g. leaving multiple loops) or other formal stuff.

Indeed it's not clearly described what exactly caused the task to stop. My impression is that this was neither caused by a stack overflow nor by a bit fault in SRAM, nor due to MISRA violations.

Last but not least, wrong behavior due to SW bugs or HW issues (e.g. CPU errata) can never be ruled out, even in the most safety relevant systems  and when following all possible processes and regulations. Therefore there has to be a safety concept on the system side which avoids critical behavior - and unintended acceleration is the top candidate there. In every ECU I ever saw there was a secondary controller checking the safety relevant paths. E.g. monitoring the pedal sensor values, comparing them to the ones reported by the CPU and shutting off the engine (or performing a reset) in case of severe misbehavior.
So it seems that Toyota (or the according supplier) hasn't implement a valid safety concept. This is the main issue, not bugs in the software or whatever.

[nctnico]

I have a problem with safety critical code and "exceptions" anyway - do you really want safety critical code to be saying "oops, didn't expect that"

Code: [Select]

if(oops_didnt_expect_that)
ShutDownEngine();
Surely better than having application data obliterated obliviously

What if this happens when your attempting to get off the train tracks, or get out of the way of a semi?  Eventually this condition will happen and the ECM must be prepared for it, exception or not.

I agree. The last thing you want is the engine to shutdown completely. My car has a protection against loading the engine too much when its cold. I had it kick in once when I wanted to overtake some cars. Very annoying and besides that the way it kicks in produces an enormous cloud of black smoke behind the car scaring the shit out of people driving behind.

[Rufus]

I partially remember reading there was a strong driver age bias (over 60 years old) to the cohort of un-commanded accelleration complainers.

I had an old lady drive into my garden a few years ago, though a brick wall. She set off from about 40 feet away and I heard a racing engine, screech of tires followed shortly by a loud crunch.

If you think you are pressing the brake and the car doesn't slow down a natural reaction is to press it harder.

[SeanB]

IIRC an exam of a TPS sensor did show the growth of tin whiskers that led to a shorting of the wiper to the one side of the pot assembly.

[Neilm]

Not helped by the "Oh we made this thing keyless so you do not have the inconvenience of needing to find the slot just press this button" who decided that you need to hold the button for 5 seconds to switch off, and also decided that if you press the button while in motion it will be disregarded as an accidental press.

I wonder what would happen if you threw the wireless key fob thingie out the window while driving...

I've not thrown one out a window, but I was driving my parents Prius which has one. The key was in Mums handbag which she picked up when she got out the car. When she was about 3 foot away the car started complaining that it could not detect the key.

[andersm]

There was an interesting link posted in the article's comments: http://www.japanfocus.org/-David-McNeill/3993

[HackedFridgeMagnet]

The Camry ETCS code was found to have 11,000 global variables. Barr described the code as “spaghetti.” Using the Cyclomatic Complexity metric, 67 functions were rated untestable (meaning they scored more than 50). The throttle angle function scored more than 100 (unmaintainable).

11,000 global variables WTF, that cant be true, I wonder if they are counting every byte in an array as a global?

Was it writen in C anyone know?

I looked up Cyclomatic Complexity, it seems an odd way to measure complexity as a switch statement with 50 cases would seem to rate as untestable. Tell me if I am wrong.

On top of that, stack-killing, MISRA-C rule-violating recursion was found in the code, and the CPU doesn't incorporate memory protection to guard against stack overflow.

Recursion doesn't seem like a good idea in embedded software. But the stack was shown to only use 94% of available, the recursion must have been limited somehow.