Audi automotive thermal imaging night vision camera teardown

[MrSquirrel]

They said it worked standalone and "provided all appropriate CAN messages" though i will ask specifically about this. English is not their main language.

[Fraser]

It was interesting to read in the BMW camera document that I referenced, that the controller effectively up-scales the camera 320x240 output to 640x480.....nice 

If the controller enabler is fully 'stand-alone', as described to MrSquirrel, and detailed on the web page, this is quite a powerful high frame rate TIC to own for a reasonable cost    Time to start hunting in the scrap yards for 'written off' high end BMW's ?    

[Lexman]

I bought the BMW variant but it is not the Pathfinder model.
Still waiting for the delivery.
It's made by Autoliv (from Sweden  ) and have 4 or 4+2 (6) pin connector.
So the night independent stand alone controller from carsystems.com will not work with this camera if it's only have 4-pin connector.
I think their second independent "solution" works only together with the Pathfinder from FLIR with 12-pin connector.
And I dont have a BMW..

Cheers
//Lexman

[Lexman]

I don't now if you have seen this video..

It's a cam from BMW E65 made by Autoliv "Autocam C1" and working as standalone.
What do you all Think about this cam ?
Looks like PathFinder from Flir.
Maybe this "old" model was not needed to be unlocked to work properly.

Cheers
//Lexman

[MrSquirrel]

It's a cam from BMW E65 made by Autoliv "Autocam C1" and working as standalone.
What do you all Think about this cam ?
Looks like PathFinder from Flir.
Maybe this "old" model was not needed to be unlocked to work properly.

Cheers
//Lexman

Yes that's the 1st generation model - this has a composite video output too so you should be able to have fun without much effort.

Edit: Some more info for you on the Gen 1 camera:

Spec sheet

http://www.flir.com/uploadedFiles/Brochure_PathFindIROEM.pdf

The PathFindIR cameras could have different firmware to the vehicle manufacturers models though? One way to find out.

The interface specification is mentioned here, but you have to sign an NDA:
http://www.flir.com/uploadedFiles/UserManual_PathFindIR.pdf?
http://cvs.flir.com/l/6132/PathFindIR-Interface-Cable-pdf/wb66k
http://cvs.flir.com/l/6132/-FLIR-PathFindIR-OEM-Cable-pdf/wb66m

So you just need a serial to LIN converter, to communicate with that camera. At an physical level you can bet they have just stuck a serial to LIN converter on top of an existing design. I wonder how similar the commands are to other Flir cores?

[MrSquirrel]

Just found this, loads of great info on how the 2nd Gen BMW system operates:

http://www.kneb.net/bmw/F01-02/06.3_F01%20BMW%20Night%20Vision%202.pdf

It is similar to the BMW manual Aurora posted, but for the 2nd gen, not 1st.

This suggests the 2nd gen camera is paired to the control unit:

Programming and Coding
Initializing Software
When replacing the camera, it is always necessary to initialize the software by entering an
enable code (FSC).

Note: The vehicle identification number (VIN) must always be entered when
ordering a new camera or a new control unit.

The camera is programmed by through of the control unit. The control unit receives the
programming data for the camera through PT-CAN. The control unit forwards this data to
the camera through the "private CAN-bus".

This is standard practise on with modern vehicles, hopefully nothing deeper? Often is simply a serial number check by a "master unit" on the bus. If the master notices one of the serial numbers (sometimes even a light cluster, or a door switch module) has changed it throws a wobbly.

However, it does not mean to say that if the end device is powered up on the bench and sent the correct CAN messages it won't come to life. I'm no expert on how BMW/Audi do it though, Honda is more my scene.

I think i would do a lot of sniffing before firing messages to either unit in case they get upset.

[Myself]

By the way, I don't know if any of the pages linked earlier in the thread have this info, but that round four-pin connector carrying the video signal is known as HSD:

http://www.rosenberger.com/en/products/automotive/hsd.php

http://www.bce.it/wp-content/uploads/2012/06/Amphenol-HSD-Connector-Introduction-BCE-06-12.pdf

http://www.te.com/en/industries/automotive/browse-products/infotainment/high-speed-data-connector-system-hsd.html

Unfortunately I don't know anywhere to source samples, but one could ask around. In the TE PDFs there's a part number for a female-female cable, 2141723, which their own site doesn't know about but Newark claims to carry it:
http://www.newark.com/te-connectivity/2141723-3/connector/dp/55W4915

All the other part numbers seem to be dead-ends, so perhaps calling one's local TE rep could yield fruit. Or just use solder -- the universal adapter!

[Lexman]

I got my BMW camera today. (Autoliv)
It was a NB2 version.
Part no: 66.54-9 205 899-01
HW:1
SW:0.1.36 / 0.0.39

I have not opened it yet but the connecting cable for the camera was included and it has ONLY 4 wires + woven shield.
The connector on the camera has 4 + 2 pins. The "extra" 2 pins is positioned on the side of the shielded round 4 pin connecter and are not used in the female cconnector (Cable side)
None of the four wires are connected to ground so I guess the neg power goes through the shield - through the shielded circular 4-wire connector.
That means we have a total of 5 wires to the camera.
I get confused ... 1 wire + shield for power, 2 wires for LVDS signal and 2 wires for the canbus = 5 wires + shield 
I have 120 ohms between two wires so they are likely the canbus.

Cheers
//Lexman

[Lexman]

Here is some pictures...

[lacatusuflorin]

I got my BMW camera today. (Autoliv)
It was a NB2 version.
Part no: 66.54-9 205 899-01
HW:1
SW:0.1.36 / 0.0.39

I have not opened it yet but the connecting cable for the camera was included and it has ONLY 4 wires + woven shield.
The connector on the camera has 4 + 2 pins. The "extra" 2 pins is positioned on the side of the shielded round 4 pin connecter and are not used in the female cconnector (Cable side)
None of the four wires are connected to ground so I guess the neg power goes through the shield - through the shielded circular 4-wire connector.
That means we have a total of 5 wires to the camera.
I get confused ... 1 wire + shield for power, 2 wires for LVDS signal and 2 wires for the canbus = 5 wires + shield 
I have 120 ohms between two wires so they are likely the canbus.

Cheers
//Lexman

The "extra" 2 pins is positioned on the side of the shielded round 4 pin connecter are use for 12 Vdc, 2 wires (red + / black -)

[mikeselectricstuff]

Do you have both ends of that cable?
From memory, the data parts are the same, but only the camera end has the power pins

[lacatusuflorin]

Do you have both ends of that cable?
From memory, the data parts are the same, but only the camera end has the power pins

I have the cable with the both end?
The end from the camera have 4+2 pins
andat  the other end have 2 terminals: 1 with 4 pins and  1 with 2 pins

From the service manual for audi the 2 pins are for power supply  with 12 Vdc

The core of this camera are similar with the FLIR TAU 2, and even have the power module of the camera similar with the tau photon replicator board for use with voltage of 12 vdc
The tau core have the posibilities to use  analog video signal NTSC / PAL, but i am not able to check the signal type for this connector.
I need only the ntsc signal from this camera to use with an external display

the 30 pins connector whwn power up have this kind of voltage levels, but i don't have an oscilloscope to see the signals forms, and i don't know how ntsc signals looks like

if you have any idea

[mikeselectricstuff]

Do you have both ends of that cable?
From memory, the data parts are the same, but only the camera end has the power pins

I have the cable with the both end?
The end from the camera have 4+2 pins
andat  the other end have 2 terminals: 1 with 4 pins and  1 with 2 pins

From the service manual for audi the 2 pins are for power supply  with 12 Vdc

The core of this camera are similar with the FLIR TAU 2, and even have the power module of the camera similar with the tau photon replicator board for use with voltage of 12 vdc
The tau core have the posibilities to use  analog video signal NTSC / PAL, but i am not able to check the signal type for this connector.
I need only the ntsc signal from this camera to use with an external display

the 30 pins connector whwn power up have this kind of voltage levels, but i don't have an oscilloscope to see the signals forms, and i don't know how ntsc signals looks like

if you have any idea

See the teardown vid - the camera itself outputs LVDS data, once you send it whatever magic commands it needs. The control box outputs composite video, but needs to think it's in a car.
I will be investigating it when I get time, but that won't be for a while

[Lexman]

I purchase the camera including its outer plastic housing. - see picture.
In the picture is the connecting cable visible, and its cut off.
The cable I have received and which were connected to the camera when I received it is also cut off but its longer than the cable in the Picture. Which explains it all.
It is not the right cable for the camera I have received. It's missing the DC power wires.
 
I have seen in the newest camera version from Autoliv (NV3), it has only 4 pin connector, the extra 2 pin for Power is missing. I wonder how the interface works in this camera. 

New info...
NV3 uses MAX9259 for the LVDS signal and the IC has also "full-duplex control channel" to control the camera. Canbus interface is no longer used.
The camera connector is 4-pin, 2-pin for Power and 2-pin för LVDS AND control.


Cheers
//Lexman

[LesioQ]

Howdy,
New here.
I also got NV2 IR cam core and would love to make good use of it. While hoping that Mike will crack the CAN init, I was wondering what B-plans can be assigned to the task.

I noticed there's an ARDUCAM pcb project, being capable to communicate with CMOS sensors. It's open to modifications, comes at $50 with QVGA LCD and snapshot storage on a card. Maybe this would be a good path to read image from ISC0601B IR sensor, once (if) we know the pinout.

Piotr.K

[TheEnd]

Hi guys,
Yes, it's someone else interested in an easy and cheap way of getting hold of a thermal imaging core with as little extra stuff tacked on.

Give me a nudge if there is another more detailed thread on this topic, otherwise I'm going to stick here as a data dump for those interested.

I have some BMW diag software, including a very basic program that will forward and return commands.

So- here are some of the possible commands and controls. Note- I think this is direct from the camera module itself!

Statuses

STATUS_KAMERA_TEMP camera temp
STATUS_HEIZUNG_TEMP heating temp (it has a heated front lens for clearing rain and condensation)
STATUS_KAMERA_SPANNUNG - spannung = voltage
STATUS_KAMERA_STUNDEN - operating hours
STATUS_SHUTTER_CYCLE -shutter cycle time in seconds
STATUS_ZEILE_LESEN - "Read data of a line from the camera.
             It is the one half of one line read (160 bytes)
             The camera image consists of 320 columns and 240 rows" - official function description
STATUS_VERDREHWINKEL - camera rotation angle, returns 1/10 deg angle
STATUS_OFFSETWINKEL - gives the offset angle, 8 bit data centred on 128decimal, and can read +/- 7.2 degrees
STATUS_DEAD_PIXEL - this returns the result of a dead pixel test, and gives the reply as a voltage(?!)
STATUS_KAMERA_BOARD - this reads ID data such as serial numbers and manufacturer codes


Controls and commands
STEUERN_TESTBILD "Displays test patterns NIVI the camera to
             Test Image 1 -> diagonal gray stripes
             Test Image 2 -> Vertical grayscale bar
             Test image 3 -> horizentale grayscale bar
             Test image 5 -> horizentale grayscale dark bottom light up"
STEUERN_SHUTTER- The following closing procedure can be set
             00 => shutter remains open
             01 => Shutter closes automatically every 2 min WITHOUT pixel calibration
             02 => Shutter closes automatically every 2 min WITH pixel calibration
             06 => shutter is closed (not Image Display)
STEUERN_WINKEL_SCHREIBEN - writes the rotation angle
STEUERN_HEIZUNG_AUS - switches heating off
STEUERN_HEIZUNG_EIN - switches heating on
STEUERN_ZOOM_AUS - zoom out
STEUERN_ZOOM_EIN - zoom in
STEUERN_TEST_PIXEL_EIN - switches on the dead pixel test, this takes 3.5 mins and outputs the total number of dead pixels - an off command is also available


Now some of the more interesting parts-
AUTHENTISIERUNG_START - for this, you need to give an "argument", in this case feed it the binary buffer, and the binary buffer is formed like this (in German, but fairly self explanatory, google translate is there if you need it!)
Als Argument wird ein vorgefuellter Binaerbuffer uebergeben
            Der Binaerbuffer hat folgenden Aufbau
            Byte 0              : Datentyp (1:Daten, 2:Maskendaten)
            Byte 1              : (unbenutzt) Wortbreite (1:Byte, 2:Word, 3:DWord)
            Byte 2              : (unbenutzt) Byteordnung (0:LSB zuerst, 1 MSB zuerst)
            Byte 3              : (unbenutzt) Adressierung (0: freie Adressierung, 1:Blockadressierung)
            Byte 4              : Authentisierungszeit in Sekunden
            Byte 5,6            : (unbenutzt) WordParameter 1 (low/high)
            Byte 7,8            : (unbenutzt) WordParameter 2 (low/high)
            Byte 9,10,11,12     : (unbenutzt) Maske (linksbuendig)
            Byte 13,14          : Anzahl Bytedaten (low/high)
            Byte 15,16          : (unbenutzt) Anzahl Wortdaten (low/high)
            Byte 17,18,19,20    : (unbenutzt) Wortadresse (low/highbyte, low/highword)
            Byte 21,....        : Schluesseldaten
            Byte 21+Anzahl Daten: ETX (0x03)


Fault code outputs
0xA788   Heater Camera
0xA789   voltage Supply Camera
0xA78A   Tamper Protection Camera uh oh! - it has a form of anti tamper
0xA78B   IR Sensor
0xA78C   Defective Pixel
0xA78D   Shutter
0xA78E   HW/SW error
0xA78F   Video out failure
0xA790   Camera not adjusted
0xA768   Camera Overtemperature
0xA769   Watching Reset (Camera)
0xFFFF   unbekannter Fehlerort


So, that might be making things overcomplicated, especially if someone is going to tap into the raw sensor data but it is explaining what the "private CAN" connections are about, and it shows the module has some brains and self diagnosis.

This particular info should be for the first version of the camera, but it looks like it applies to all of them.

I haven't got a camera, I'm not on the level of the hardware hacking that some people might be on, but if i can help, I will!

[LesioQ]

Hi,

That indeed sounds promising !
Do these commands have any hex representation ? Do You have a command id table or something useful when trying to communicate with bare cam via CAN ?
Piotr.K

[TheEnd]

Sadly it's doesn't go that far, the program is still a "dashboard" or front page, and the data will be interpreted and formed into a CAN command somewhere else.

Another translation table shows-
NR   MODE   MODE_TEXT
0x81   DEFAULT   DefaultMode
0x82   PT   PeriodicTransmissions
0x84   EOLSSM   EndOfLineSystemSupplierMode
0x85   ECUPM   ECUProgrammingMode
0x86   ECUDM   ECUDevelopmentMode
0x87   ECUAM   ECUAdjustmentMode
0x88   ECUVCM   ECUVariantCodingMode
0x89   ECUSM   ECUSafetyMode
0xFA   SSS_A   SystemSupplierSpecific (A)
0xFB   SSS_B   SystemSupplierSpecific (B)
0xFC   SSS_C   SystemSupplierSpecific (C)
0xFD   SSS_D   SystemSupplierSpecific (D)
0xFE   SSS_E   SystemSupplierSpecific (E)
0xXY   --   unbekannter Diagnose-Mode

So it looks like there is a command to ask what mode it is in, and the hex answer will define what it comes back as saying.
This could well be a standard table for all modules in the cars, but it might also mean it could be set to a development mode 0x86, if the command to change modes could be found.
I don't know much about CAN and the data side of things, but I'm guessing it'll make more sense to others.
If it's useful, maybe it'll mean a way to get into these devices, if it isn't, well, hey, you can never have too much information.

[LesioQ]

Meanwhile I noticed that my Audi NV2 responds to 0x301 to 0x303 frames (remote type) at 100kbps.
Does anyone know IF I can use CAN interface to talk to LIN device ? (BMW NV1 cam).
Piotr.K

[yogort1]

Anyone managed to run this camera?

[ogoun]

Greets all,

Has anyone got any further with this? I have both the cam and the controller, and have disassembled both, and have the serial eeprom from the camera (used with the Xilinx spartan3e FPGA). I will look at reading it (after making an adapter), to see if anything can be gleaned from its contents, as I suspect that the spartan3e has one of the Xilinx microcontroller cores in it, possibly a microblaze. If it does, then there may be some areas of the eeprom that have the code for this micro core. If so, it may shed some light on how the private can bus is used. Perhaps this will be enough to learn how to "wake up" the cam.

Cheers,
Pete

[ogoun]

Some interesting patents:

7470902 http://www.freepatentsonline.com/7470902.pdf

6812465 http://www.freepatentsonline.com/6812465.pdf

6028309 http://www.freepatentsonline.com/6028309.pdf

Gives a bit more info on the camera module and the FPA (320x240 focal plane array) chip

Cheers,

Pete

[therew]

Does automotive thermal imaging have something in common with thermal systems for tactical/hunting purposes? I have a couple of the finest thermal imaging scopes from http://www.atncorp.com/thermal-night-vision and I was wondering whether this technology is similar to what I can install on a car. My father got a FLIR thermal imaging night vision camera for a 99-03 Cadilac Deville, installed a 5" monitor above his radio and installed the camera behind the grill. Looks awesome.

[encryptededdy]

Does automotive thermal imaging have something in common with thermal systems for tactical/hunting purposes? I have a couple of the finest thermal imaging scopes from http://www.atncorp.com/thermal-night-vision and I was wondering whether this technology is similar to what I can install on a car. My father got a FLIR thermal imaging night vision camera for a 99-03 Cadilac Deville, installed a 5" monitor above his radio and installed the camera behind the grill. Looks awesome.

I'm not sure about the new ATN thermal sights (the one with WiFi), however their older ones used FLIR Tau2 cores which is very similar if not the same sensor wise compared to automotive systems like the FLIR PathfindIR

[nixxon]

See the teardown vid - the camera itself outputs LVDS data, once you send it whatever magic commands it needs. The control box outputs composite video, but needs to think it's in a car.
I will be investigating it when I get time, but that won't be for a while

Have you had a chance to investigate the Audi Night Vision any further, Mike?