Audi automotive thermal imaging night vision camera teardown

[mikeselectricstuff]

[mos6502]

Here's an old thread on a German forum that deals with a BMW system:

http://www.mikrocontroller.net/topic/177639

If you scroll down for the pictures, you can see that the sensor module looks similar to yours. It's different inside, though. Different sensor, Altera FPGA.

Anyway, the upshot seems to be that at least the BMW system, the sensor module, the control module and the car's ECU are locked together. They do a little chat on powerup and exchange serial numbers and stuff. This prevents the camera from being stolen and used in a different car, even if it's the same model.

Here's an old listing on eBay for an "unlocking module":

http://www.ebay.de/itm/ws/eBayISAPI.dll?ViewItem&item=330774553378&ssPageName=ADME:B:SS:DE:1123

It connects as a man in the middle device between the car's CAN bus and the control module's CAN bus.

What kind of data does the actual sensor itself spit out? Would it be feasible to just hook it up to an FPGA board?

[Fraser]

Mike,

You never cease to impress !

Thank you 

[amyk]

Wow... is this the fourth or fifth one he's taken apart now?

If I remember correctly from reading some documents Motorola "accidentally" exposed many years ago, the 9S12 can be powered up in a special internal peripheral testing mode where it's possible to unsecure the flash and thus readout the firmware.

[Stonent]

Next up on Mikeselectricstuff, Audi FLIR module connected to 1000 ipod displays mounted in a public place.

 

[mikeselectricstuff]

What kind of data does the actual sensor itself spit out? Would it be feasible to just hook it up to an FPGA board?

I would expect it to be similar to the E4 sensor, but at the moment something is stopping the sensor doing anything.

[Stonent]

What kind of data does the actual sensor itself spit out? Would it be feasible to just hook it up to an FPGA board?

I would expect it to be similar to the E4 sensor, but at the moment something is stopping the sensor doing anything.

It doesn't by chance have a similar pin configuration to the E4 does it? Perhaps you could wake it up using the E4 as a host.

[mikeselectricstuff]

What kind of data does the actual sensor itself spit out? Would it be feasible to just hook it up to an FPGA board?

I would expect it to be similar to the E4 sensor, but at the moment something is stopping the sensor doing anything.

It doesn't by chance have a similar pin configuration to the E4 does it? Perhaps you could wake it up using the E4 as a host.

No totally different - When I get time I will try throwing a bunch of CAN at it to see if I can get it to run. Worst-case I may have to rent an i3 off ebay to see what the i-series sensor needs to be driven with.

[mikeselectricstuff]

I suspect most of the protection is in the control unit, and the cam itself may just need a command to turn on - I can see some line/frame sync type signals in the serialised steam, so there's probably just s 'go' signal somewhere.
They _could_ have implemented some nasty challenge-response authentication though. The only obvious NV memory is the config flash  - not looked hard enough yet to determine if one of the 8 or 6 pin devices is an eeprom.

[amyk]

http://cvs.flir.com/tau2-electrical-idd
Does that 50-pin connector look familiar...?

http://cvs.flir.com/tau2-product-spec
This document says that all the outputs can be set to various options, one of which is "disabled". Maybe the camera module has just been configured with all the outputs disabled by default?

http://cvs.flir.com/tau2-quark-software-idd
Here details the commands that are used to change the settings, including the outputs.

[mikeselectricstuff]

http://cvs.flir.com/tau2-electrical-idd
Does that 50-pin connector look familiar...?

http://cvs.flir.com/tau2-product-spec
This document says that all the outputs can be set to various options, one of which is "disabled". Maybe the camera module has just been configured with all the outputs disabled by default?

http://cvs.flir.com/tau2-quark-software-idd
Here details the commands that are used to change the settings, including the outputs.

No - the connector is smalley. Look slike it has 4 LVDS pairs plus a few other oddments. Two of the LVDS likes look like clock and a datastream, but only sync present. I will take a look at the serial flash - it will be pretty obvious is there's anything in there apart from the FPGA bitstream.
It is also possible there's an eeprom - again won't be hard to find.

[Rasz]

How much was it Mike? I remember looking them up when you did E4 teardown and people wanted ridiculous amounts of money for scrapped Audi cameras (>$1000).

I would expect it to be similar to the E4 sensor, but at the moment something is stopping the sensor doing anything.

I dont think there is anything stopping it from working, I think its the other way around - FPGA didnt initialize the sensor yet and is waiting for the command from the head unit.

Those Flir sensors look to be designed just like cmos camera sensors, with buildin ADC and all. I suspect there is I2C interface to the ADC inside and its waiting for commands (gain,exposure,start,stop). Without those commands it just sits in idle state.

We need ISC0601B pinout.

You did probe around E4 sensor, was there something resembling i2c bus on the sensor pins? Does changing levels in the camera change gain in the sensor? or is it fixed 16 bit output leaving all this stuff to camera cpu/fpga for postprocessing?

[mikeselectricstuff]

How much was it Mike? I remember looking them up when you did E4 teardown and people wanted ridiculous amounts of money for scrapped Audi cameras (>$1000).

There's tons of them on Ebay.de, around the 500-600 Euro mark

I dont think there is anything stopping it from working, I think its the other way around - FPGA didnt initialize the sensor yet and is waiting for the command from the head unit.

Those Flir sensors look to be designed just like cmos camera sensors, with buildin ADC and all. I suspect there is I2C interface to the ADC inside and its waiting for commands (gain,exposure,start,stop). Without those commands it just sits in idle state.

We need ISC0601B pinout.

You did probe around E4 sensor, was there something resembling i2c bus on the sensor pins? Does changing levels in the camera change gain in the sensor? or is it fixed 16 bit output leaving all this stuff to camera cpu/fpga for postprocessing?

Not probed the E4 sensor yet as it's hard to access - need to make a riser/breakout - I've ID'd the connector but couldn't find any stock of both parts.

[equinoxe]

Here's an old thread on a German forum that deals with a BMW system:

http://www.mikrocontroller.net/topic/177639

If you scroll down for the pictures, you can see that the sensor module looks similar to yours. It's different inside, though. Different sensor, Altera FPGA.

Anyway, the upshot seems to be that at least the BMW system, the sensor module, the control module and the car's ECU are locked together. They do a little chat on powerup and exchange serial numbers and stuff. This prevents the camera from being stolen and used in a different car, even if it's the same model.

Here's an old listing on eBay for an "unlocking module":

http://www.ebay.de/itm/ws/eBayISAPI.dll?ViewItem&item=330774553378&ssPageName=ADME:B:SS:DE:1123

It connects as a man in the middle device between the car's CAN bus and the control module's CAN bus.

What kind of data does the actual sensor itself spit out? Would it be feasible to just hook it up to an FPGA board?

The module is also for sale here:
http://www.navi-tools.de/index.php?option=com_content&view=article&id=4&Itemid=5&lang=en
Sadly they don't really give any useful info (probably to protect their business).

[mikeselectricstuff]

Here's an old thread on a German forum that deals with a BMW system:

http://www.mikrocontroller.net/topic/177639

If you scroll down for the pictures, you can see that the sensor module looks similar to yours. It's different inside, though. Different sensor, Altera FPGA.

Anyway, the upshot seems to be that at least the BMW system, the sensor module, the control module and the car's ECU are locked together. They do a little chat on powerup and exchange serial numbers and stuff. This prevents the camera from being stolen and used in a different car, even if it's the same model.

Here's an old listing on eBay for an "unlocking module":

http://www.ebay.de/itm/ws/eBayISAPI.dll?ViewItem&item=330774553378&ssPageName=ADME:B:SS:DE:1123

It connects as a man in the middle device between the car's CAN bus and the control module's CAN bus.

What kind of data does the actual sensor itself spit out? Would it be feasible to just hook it up to an FPGA board?

The module is also for sale here:
http://www.navi-tools.de/index.php?option=com_content&view=article&id=4&Itemid=5&lang=en
Sadly they don't really give any useful info (probably to protect their business).

..well that probably rules out a complex enabling protocol - unfortunately I don't have access to a system installed in a car. When I get time I'll take a look at the CAN traffic & try throwing some commands at it

[MrSquirrel]

Great teardown - this is very interesting, think i'm going to have to get one of these myself.

I noticed the PCB had the marking "NV2":
NV2 = Second version of Autoliv Night Vision with a 24 degree Field of view, 320x240 pixels, 25u pitch.
According to some sales presentation stuff i found the camera is around $800 and image sensor alone $500. Ouch!

http://www.hanser-automotive.de/fileadmin/heftarchiv/2004/17700.pdf
Interesting block diagram on page 1. Camera simply pumps out raw video stream with CAN for control. Means camera module is somewhat universal, and processing engine can vary depending on that the OEM wants to do with it.

Looks like these guys had a go:
http://www.nightvisionforums.com/phpBB3/viewtopic.php?f=1&t=6799

Info on NV2 system, product roadmap etc:
http://www.chalmers.se/safer/EN/news/events/night-vision/downloadFile/attachment/Slides_Night_vision_presentation?nocache=1226396024.19

Teardown of the 1st Gen System
http://www.xolmatic.com/xprojects/XE65/Night_Vision_Camera.htm

More great 1st Gen info here, pics of 3rd Gen (NV3) board at bottom of page?: (oops, already posted earlier by mos6502)
http://www.mikrocontroller.net/topic/177639

And some tasty stuff here:
http://liu.diva-portal.org/smash/get/diva2:218938/FULLTEXT01
It is actually quite an interesting read in it's own right.

This document references a document called:
"Sundin Mats (2006) NV2 LVDS video link specification, Linköping: Autoliv Electronics AB"

Looks like that is the jackpot, but i can't find a copy. Perhaps e-mail the guys that wrote the thesis. 

[Rasz]

nice find, but as you probably noticed in teh pdf it relies on a car initializing the whole system
intercepting lvds data is "easy" (can read it before/after ldvs bridge chip) compared to guessing secret handshake headunit might have with ecu, or randomly fuzzing canbus between camera and headunit in hopes of initializing camera
This is the problem Mike has now.

Its probably as simple as headunit sending "#camera_address GO" to the camera unit.
Might be easier to just connect bare sensor to FPGA.

[amyk]

This document references a document called:
"Sundin Mats (2006) NV2 LVDS video link specification, Linköping: Autoliv Electronics AB"

Looks like that is the jackpot, but i can't find a copy. Perhaps e-mail the guys that wrote the thesis. 

Notice the phrase "video link" - we already know the video data format so probably nothing interesting in there that isn't already covered in Flir's public datasheets. A control link spec on the other hand...

[Fraser]

Slightly OT

An interesting document from Autoliv on the challenges of providing LW thermal cameras for the masses. Not applicable to the teardown specifically, but interesting to see what AutoLiv are up to and the hopes of the future. The lowering of the sensor and optics costs is detailed.....seems like the Flir E4 strategy.

http://www.autoliv.com/ProductsAndInnovations/Documents/Research%20Papers/4.%20Autoliv%20Research.pdf

[yogort1]

Anyone found out what this secret handshake consists of?    With this information it would be possible to make some fairly simple startup emulator without having to spend 250E for activator module 

[Rasz]

http://youtu.be/7h7LWeET1fI?t=19m01s

might work for powering this camera up

[Lexman]

Hello everyone !
Great teardown and very interesting project. I have also order one cam from ebay.
I hope we can make some progress with this camera.
It will probably take a week or so until I got my camera.
I do not know if it's a NV1 or NV2. I could not see this in the pictures.
BTW I think we have to send a request via the CAN bus for each individual frame ... if I understand the docs. correct.

Cheers
//Lexman

[MrSquirrel]

Did you buy the Audi or BMW variant?

I e-mailed these guys about their "enabler" unit for the BMW version:
http://carsystems.com.ua/en/intelligent-modules.html

Connection diagram attached for the NV1 and NV2 version. Looks like the vehicle only has basic integration with the control unit.

Unfortunatley i can't find anyone doing an equivalent for the Audi model, i wonder how similar the protocols are?

They want $250 for their module. Pretty reasonable as it potentially saves a lot of work and you can re-sell both it and the control unit on when the camera protocol is understood.

The BMW system seems to attract far higher prices on eBay, a bit too high IMHO for playing with. Audi units are a lot more available and realistically priced.

[Fraser]

IIRC the camera enabler only 'fools' the camera system controller into accepting the control commands from a genuine BMW central command bus. It cannot be used to fool the camera controller as a stand alone unit or to make a BMW camera work in an Audi etc.

Edit: This unit appears to be able to work in stand alone mode so I may be wrong about it still needing the BMW CAN Bus:

http://carsystems.com.ua/en/intelligent-modules.html

Definitely described as 'stand-alone' yet I read a reply from the company previously (In answer to a question about using it in another brand of car) and they said it would only work on a BMW of the correct model. Maybe they have improved it ?

[Fraser]

If this has already been found, apologies. I attach an interesting document detailing the BMW system. Found here:

http://www.bimmerboard.com/members/jcarey/original/BMW%20Night%20Vision.pdf