Toyota firmware fail

[0xdeadbeef]

11,000 global variables WTF, that cant be true, I wonder if they are counting every byte in an array as a global?

This is typical in automotive software. The calibration guys want to look at every internal variable for validation and calibration. To be able to export variables to the calibration tool, they have to be defined global as statics are not listed in the map file (and might be decorated in the debug info). So global (in C sense) doesn't necessarily mean that these variables were really used globally. Then again, using global variables as interfaces between functionalities is unfortunately also pretty common in automotive software - at least in the higher levels. Partly because it's very effective and partly because all the global variables exist anyway (for calibration).

Was it writen in C anyone know?

I would think so. Automotive software is usually either written in C or autocoded (resulting in C code again).

Recursion doesn't seem like a good idea in embedded software. But the stack was shown to only use 94% of available, the recursion must have been limited somehow.

Yeah, recursion would be pretty bad. Even though recursion should not mean "endless recursion". Anyway, you'd need to see the exact code to judge how bad this really was. MISRA violation says nothing at all.

[N TYPE]

I'm pretty sure it's a Japanese company DENSO who handles all of the ecu's and electronics for Toyota. they have done so for decades.

[N TYPE]

I'm pretty sure it's a Japanese company DENSO who handles all of the ecu's and electronics for Toyota. they have done so for decades.

Having said that, DENSO are responsible for the Japanese Domestic market. They normally use DENSO branded chips which are typically just customized brand-name micro's, restamped with denso's logo.
 I notice this particular case its from an American Prius which the ECU could have been developed by the americans to meet their own emissions standards and safety laws etc etc, considering they're using NEC parts rather than DENSO, it wouldnt surprise me if this were the case.

[G0HZU]

1. Toyota is a car manufacturer, they do not (to my knowledge) design and manufacture car electronics.
The real supplier of the ECU is not named - strange.

Google for NipponDenso or just Denso. They are owned by Toyota and have been making engine management systems for Toyota vehicles for decades.

I can't say if the ECU in question was made by ND but I can say that I have a LOT of experience in reverse engineering Toyota ECUs of the 1980s and 1990s and they were all made by Denso. Note: The image of the ECU in the article looks like it has a Denso 'D' logo on one of the chips but this logo is newer than the one I'm used to seeing so I can't be certain.

One thing that impressed me a lot about Denso ECUs of the 80s and 90s was the attention to detail in terms of safety and also the quality and efficiency of the coding.

You could also access memory addresses via a simple serial interface which allowed the user to log all aspects of the ECU whilst it was being developed. So looking at the memory used /not used by the stack was possible whilst it was being driven. Also all of the variables and status flags could be logged with several hundred variables logged a second.

I learned an awful lot about efficient coding from analysing Denso's code. I guess that modern ECUs will be far more complex and will be written in a high level language so bugs are far more likely. I looked at dozens of Denso ECUs and only ever found a couple of minor bugs.

[nctnico]

Until the early 2000's Toyota's cars where rock solid. After that it went downhill. For example they have had huge problems with their diesel engines which got so bad that Toyota buys diesel engines from BMW nowadays. Picture this: the biggest car manufacturer in the world suddenly can't design a reliable diesel engine. A reliable/durable diesel engine was one reason to buy a Toyota for many people. I guess at some point the good people in the engineering department left or retired.

[Stonent]

I think Continental and Siemens are some ecu brands I've seen in vehicles that were labeled.  Many just have the car brand.

[0xdeadbeef]

Siemens Automotive was bought by Continental some years ago (which didn't make ECUs before). So they're essentially one brand regarding ECUs.
Other big players are Bosch, Delphi and Magneti Marelli.

[Stonent]

Siemens Automotive was bought by Continental some years ago (which didn't make ECUs before). So they're essentially one brand regarding ECUs.

The first time I saw that I thought "Don't they make tires?"

[0xdeadbeef]

The first time I saw that I thought "Don't they make tires?"

They do But they had a much larger product range even before they bought Siemens VDO Automotive.

[Andreas]

I once drove a (company) VW with an automatic gearbox in which there was a short between the brake lights and the normal lights. It made the car run like the fuel tank was as good as empty if/when I turned on the head lamps.

Congratulations: you have successfully tested one of the features of the monitoring concept.
If brake is pressed (brake light switch active) while the accelerator is pressed it is assumed
that the accelerator spring is broken and the pedal stucks.
In this case engine torque is reduced by the brake signal.

4. All Tier 1 suppliers for OEMs have to follow strictly on design processes for Hardware and Software, which are standard in the automotive industry,

The processes are the one, but the opinions on how a car has to work may be different in other countries.
Do not forget: here in Germany (and maybe Europe) due to the VDA guidelines ("EGAS Lastenheft") we have
a relative good standard regarding UA in hardware and in software.

http://www.iav.com/sites/default/files/attachments/seite/ak-egas-v5-5-en-130705.pdf


But often hardware restrictions and driver behaviour may compromize the standards.
Just 2 (of course freely invented) examples:
1.  a car manufacturer wants to have implemented a cruise control software in a elder generation ECU for a model year upgrade. But the car has no redundant brake switch which is mandatory (according to VDA) for this. But how to convince the overseas manufacturer to implement a better break pedal (and of course a new cable harness).

 I don´t know how this would end. But probably the manufacturer signs a weaver and takes the responsibility so the Tier1 will deliver a ECU with cruise control for only 1 brake switch.

2. There may be different opinions between american car manufacturers and european deliverers regarding "two footed driving"

At least if I believe a EE of a american car manufacturer the normal way to drive a american car is to put the right foot on the accelerator and the left on the brake pedal (with automated gear). The brake light switch might be always on during this procedure. And maybe the redundant brake switch is still unactivated due to switch tolerances leading to plausibility error between the switches.

From monitoring side engine torque and cruise control have to be switched off when only one switch is activated. (Because the other could be defective). Further depending on debouncing strategy after some debouncing time or some of those implausibility events a fault code will be entered in fault memory switching all comfort functions off.
From OEM side this behavior is unwanted regarding two footed driving.
So will the OEM spend more money for a additional brake pressure sensor?
Or even a better pedal with analog sensors like in brake assistant?

Other stuff like the paradigm of "mirroring all important data" is questionable at least. Apart from the runtime hit when mirroring all important variables and consistency issues: what should you do if the main value and the mirrored value differ? Which one would you trust?

According to the VDA rules there is much more than just only mirroring important data.
You have a 3-Level concept:

Level 1 is the standard application software with the normal torque path. Usually no mirrroring of data.

Level 2 is the functional monitoring software with a simplified model of the torque path.
If the torque in level 2 is lower than in level 1  (with some percent offset) then the torque in level 1 is limited.
In Level 2 each variable has a cyclical RAM check and either a complementary storage or a checksum.
The question which one is to trust is simple: neither. The ECU does the same what you would do with a hanging computer: simply press the reset button. If the error persists either ECU or external watchdog will shut off all power stages finally.

Level 2' is the same as Level 2 but in this case fixed sensor values out of arrays are used for calculation (giving predefined final results) to prove that the processors ALU is still calculating in a right manner. The result is not compared directly but fed together with other tests to the final response for the external watchdog.

Level 3 monitors the hardware and the external watchdog. The watchdog asks questions to the CPU and shuts down the power stages if  the response is either wrong or outside of the time window. The response is calculated as result which depends on the execution and order of the level 2 routines.

With best regards

Andreas


 

[Dr. Frank]

The first time I saw that I thought "Don't they make tires?"

They do But they had a much larger product range even before they bought Siemens VDO Automotive.

That's right. We have undergone many steps of metamorphosis in the past 15 years:

It started with VDO (instrument clusters /*/ and other electronics).
Then we became Mannesmann - (seamless steel pipes)  VDO + Kienzle (electronics / mechanics for vans + mobile license) + Philips Car Systems (radio + navigation).

Then the ww. stock bubble was initiated in 1999, when a much smaller company (for the first time ever) bought Mannesmann: Vodafone wanted to acquire the mobile license.

So we belonged a short time to vodafone, being named ATECS.

Shortly after that, we were sold to Siemens Automotive.

VDO and Siemens both made ECUs already.

Then, Continental (tyres) bought SiemensVDO in 2007, but they already had : Teves (brake systems), Motorola Automotive (telematics) and ContiTech (rubber belts).

Then in 2008 Schaeffler KG (ball-bearings) tried to take over Conti, but in the end, they suffered greatly from the finance crisis in 2009/2010 and had to sell again parts of their shares.

So, without changing job, in the last 16 years at VDO we had at least 6 different owners or company names, and the number of people increased from about 15.000 to now over 175.000. Some times, it was quite an uncomfortable situation for the people, due to many changes, but it was also very exciting to experience / take part in all those business migrations.


Well, from the photograph in the EDN article, the Toyota engine control electronics perhaps is produced by Delphi, as they call theirs ECM also, and it's called ECU otherwise.

http://delphi.com/manufacturers/auto/powertrain/gas/ecm/
Those units also feature ETC.

And this company was already mentioned in connection with this Toyota incident.

Also, I did not see the typical Conti imprint or code numbers on the PCB.

So, I feel relieved that it's obviously not our company to blame, as Toyota will for sure charge the supplier for the 3M$, if the SW really is as badly engineered as described (which I really doubt at the moment).
For sure there will also be a big mess for the complete series production, as this judgement will cause further trials in US, further compensation payments, and worst case a massive call-back.

If latter event happens, this might ruin the supplier.

Frank


/*/ 111 years of Tachometer

[0xdeadbeef]

According to the VDA rules there is much more than just only mirroring important data.
You have a 3-Level concept:

Yeah, but the article didn't talk of a safety concept at all but claimed that every "important" data - including RTOS task activation bits - should be mirrored.
This is nonsense in my point of view as it completely misses the point.

[Andreas]

Yeah, but the article didn't talk of a safety concept at all but claimed that every "important" data - including RTOS task activation bits - should be mirrored.

Ok in this case you are right. You will have to prove that the task (or at least the safety relevant routine) is actually activated. Usually this is done by cryptographic methods when entering the safety relevant code and exiting it.
If something goes wrong then the answer to the external watchdog will be wrong. The mirroring of the activation bits is useless if the task is activated but terminated on its way.

But I fear that in the according ECU no external challenge response windowed watchdog is used.

With best regards

Andreas

[0xdeadbeef]

That's why I said that if there is a flaw in the ECU, it would be mainly a flaw in the safety concept.
Letting the CPU monitor itself has only limited use IMHO. It's important to have a bulletproof safety concept on the system side. Starting from mechanics (e.g. throttle blade going to limp home position if not controlled, dual sensors for pedal and throttle position) over hardware (e.g. shutdown paths in HW, secondary CPU monitoring the main CPU) up to the application software. Anyway, the software can be only a part of the concept, not the only place to perform safety checks. After all: if the CPU stopped to work correctly, skips tasks, has corrupted stack etc: who could be sure that the monitoring functions in the software would work at all?

[qno]

  This is the US.
If you ask a company that sells software analysis to analyse your software the will find something.
Fortunately the ECU does not run on Windows. Else you will get an update every 2 weeks.

[G0HZU]

Well, from the photograph in the EDN article, the Toyota engine control electronics perhaps is produced by Delphi, as they call theirs ECM also, and it's called ECU otherwise.

http://delphi.com/manufacturers/auto/powertrain/gas/ecm/
Those units also feature ETC.

And this company was already mentioned in connection with this Toyota incident.

Also, I did not see the typical Conti imprint or code numbers on the PCB.

Hi Dr Frank
Did you miss my earlier post?
I'm virtually certain that it's a Denso ECU. Denso are owned by Toyota and they have been making ECUs for Toyota for many, many years.
 
Two of the chips on it have the newer Denso logo on them and if you do what I suggested earlier and google "Denso" then you will see a very similar ECU circuit board on their European website.

I spent several years taking various 1980s and 1990s Denso ECUs apart and reverse engineering their program code and I was always impressed with the quality of their ECUs.

For example, the ECUs for the UK market were designed to be as failsafe as possible. Even if the main MCU chip fails the ECU can fall back into analogue mode and keep sending basic ignition pulses and signals to the fuel injectors to try and keep the engine running well enough to allow the driver to have some control as this buys vital time to look for somewhere safe to stop.

All the time this is happening the ECU also tries to restart the MCU chip via a hard reset loop.

They were masters of efficient coding as well and only had a few kb of ROM to play with. So one trick to save space was to program at hex level allowing the MCU to jump into the middle byte of two byte instructions as a way of changing the way the program runs.

This gives a disassembler a tough time because this is illegal to most disassemblers. I wrote my own module for IDA Pro to cope with the 1990s ECUs as they also had a unique instruction set that took a fair bit of reverse engineering! (no datasheet or instruction set is available for the MCU)

[G0HZU]

Quotes from the EDN article:

Toyota claimed the 2005 Camry's main CPU had error detecting and correcting (EDAC) RAM. It didn't. EDAC, or at least parity RAM, is relatively easy and low-cost insurance for safety-critical systems.

I can only speak for the (fairly basic) 1980s and 1990s ECUs from Denso and these had basic error detection on the battery backed up RAM.

Every few milliseconds during ECU runtime the ECU runs an error/health check on these variables to see if they ever get corrupted as it stores this data in more than one way. I think the ECU alerts the driver with a dash lamp if there is a problem detected.

Most of the ECUs also ran checksums on the code and also basic read/write health checks on the entire system RAM during the initial few milliseconds after the ignition was turned on at the key.

They also failed to perform run-time stack monitoring.

This could be (crudely) done on the 80s and 90s Denso ECUs because I did it myself via their inbuilt (hidden) diagnostics port inside the ECU.

Two key items were not mirrored: The RTOS' critical internal data structures; and—the most important bytes of all, the final result of all this firmware—the TargetThrottleAngle global variable.

The Denso ECUs I looked at were able to read the TPS or throttle position many, many times per second so I'm not sure 'why' anyone would want to mirror this data.

If it read it or stored it wrongly in one instance then it would get refreshed in the blink of an eye anyway. The old ECUs also had the ability to detect 'untypical' results from the TPS and reject them (and store an error code and alert the driver instantly via the dashboard)

[G0HZU]

I looked on the web for more info on the affected ECUs and the general thought is that some 'task' carried out by the controller gets corrupted and doesn't finish and this corrupts the stack causing mayhem.

On the older Denso units the ECU had various healthchecks to guard against a system crash. Obviously this was a far simpler system but the ECU (and the MCU) was able to detect a system crash and reboot the MCU within a fraction of a second. It's a few years since I looked at this stuff but the ECU and the MCU worked together to produce a kind of 'analogue + MCU' heartbeat and if this missed a beat then the MCU was rebooted and the failsafe system kicks in to keep the injectors and ignition running in the very crude analogue override mode. Hopefully, the MCU then reboots in a fraction of a second and the MCU then regains control and runs OK for the rest of the journey.

Denso even went as far as filling all unused bytes in the ROM with a special instruction opcode that rapidly forced a reboot if the MCU ran off the rails into uncharted/unused parts of its ROM memory.

The ECU boot up code also flushes the RAM and writes sensible values into RAM for the initial lap around the main program loop until the ECU has read all the sensors and found its feet.

I suppose one potential bug on a fly by wire throttle could be if the ECU MCU kept crashing/rebooting due to a stack or RAM failure at some point in the overall program. It would therefore keep reusing a fixed initial default throttle setting during each attempted reboot up until the program crashed again (before getting to respond to the throttle position set by the driver). However, I would seriously hope that Denso would not make such a mistake in the reboot code.

[dannyf]

This article gives a laundry list of reasons why the code could have failed, but apparently no one knows if or how it did, so what's the point?  It's a legal rather than technical discussion.

You nailed it right there.

This is a war funded by lawyers, fought by lawyers and will be won / lost by lawyers too.

All it does is is to add costs to future vehicles that innocent consumers pay.

[G0HZU]

Having said that, DENSO are responsible for the Japanese Domestic market. They normally use DENSO branded chips which are typically just customized brand-name micro's, restamped with denso's logo.
 I notice this particular case its from an American Prius which the ECU could have been developed by the americans to meet their own emissions standards and safety laws etc etc, considering they're using NEC parts rather than DENSO, it wouldnt surprise me if this were the case.

I looked at dozens of 80s and 90s Toyota ECUs from UKDM cars, USDM cars and JapDM. All were made by Denso

The MCU chips they used were usually special versions of mainstream MCU chips that had extra features that weren't on the mainstream equivalent MCUs.

This was probably for copy protection as well as for custom requirements. Also Denso made their own custom IC 'modules' that had thick film technology inside. I think this saved space on the PCB and also provided a degree of copy protection.

I worked out how the MCU operated and was able to make the ECU run my own copy of the factory ROM so I could modify it at will. eg I partly rewrote their code to support real time map retuning by uploading a version of factory code with the maps running in (external) RAM rather than internal ROM and I upgraded their diagnostics to include write access rather than just read access.

So it could still datalog but it could also show where it was on its maps as well as allow retuning via a laptop on a rolling road

[Teemo]

The complete design should be made public to make the evaluation possible    It should be basic right of everybody to examine the safety related design of the appatatus they are using.

If the design is not made public or is presented in unverifyable (uncomplete , unsystematic) manner, the one should assume that it is unadequate, and not to use the apparatus.

[Dr. Frank]

Hi Dr Frank
Did you miss my earlier post?
I'm virtually certain that it's a Denso ECU. Denso are owned by Toyota and they have been making ECUs for Toyota for many, many years.

Hi,

well yes, I just read your previous post again: "I can't say if the ECU in question was made by ND..." and you were not certain about the logo...

Anyhow, if it's really Toyotas internal supplier, or anybody else, I still can't believe that Toyota should have accepted bad SW engineering (that's the key argument of the so called embedded experts).

And anyhow, that judgement opened the door for further costly trouble for Toyota in US, and that's right, it's all about legal arguments now, not technically anymore.

Frank

[G0HZU]

it's all about legal arguments now, not technically anymore.

Yes, it's certainly getting scary for Toyota...

However, here's a link to the Denso website that shows a similar ECU

http://denso-europe.com/products/electronics/

It still doesn't conclusively prove that Denso make the 'whole' ECU but I think it's safe to assume they are heavily involved in this case.

I also managed to find a listing of the court case proceedings online and you can see the embedded expert (Mr Barr) getting cross examined by both sides. There's also reference to Denso being involved in the ECU design and production.

I only skimmed through the court proceedings but it seems like Barr tried to invoke the UA (unintended acceleration) by deliberately flipping bits in registers/memory during roadtests. Toyota gave him help in setting up the test with a system that let him corrupt the memory in real time in an attempt to invoke the UA.

It appears that in every attempt he failed...

However, Toyota still lost the case and I hope I never have to be involved in a legal case in a US court.

[dannyf]

It still doesn't conclusively prove that Denso make the 'whole' ECU but I think it's safe to assume they are heavily involved in this case.

I would be surprised if Denso is a major player in the ecu (chip) business: Renesas and Freescale are two names I would put on top in that market.

However, Toyota still lost the case

Our legal system relies heavily on who has the best lawyers (who can spin) than who has the facts on his side.

[G0HZU]

Our legal system relies heavily on who has the best lawyers (who can spin) than who has the facts on his side.

Here in the UK I think the legal system is slightly less extreme but still very unimpressive...

According to the court notes Barr earned about $500 per hour over thousands of hours on the case so he definitely 'won' from a financial point of view. So I see him as being competent at making money and writing books about coding standards and saying how bad Toyota/Denso are at coding in a court.

But maybe a lot less impressive at breaking their product in the manner he says they can be broken

Why did he fail to invoke the UA if Toyota's poor coding standards code broke so many violations?

He was even allowed to spend thousands of hours studying the code (with Toyota to help him) and could flip bits illegally to try to make the ECU go into the UA mode. But from my initial reading of the court notes he appears to have failed.