Autoliv NV2 on Raspberry Pi

[Treehouseman]

Got a bit more progress, I'm really starting to fight signal loss, I'm going to have to rework the connection between the deserializer and the camera, I'm not sure what makes the difference between a good capture and a bad one, without touching the camera I can get a good 30 second capture and then instantly running a second one get maybe one or two good frames out of the entire thing.

[LesioQ]

Is this video without any shutter operation ? Not bad if without calibrations.

[Treehouseman]

The shutter isn't really used much here, but you can see it actuate in the first capture in the video, there's some "blank" frames but it you look at it frame by frame you can see the shutter sweep in and out. If you look at a Flir E4 or Seek as a "photometric" camera this is a camcorder. This doesn't supply metered values or at least it's not intended to. It may be possible to "calibrate" it's output and tell what temp something it but it's intended to only show differences in heat, it's only used as a people/animal/car detector at night to show things out of range of your headlights. The shutter does actuate on this but I don't know if it's the sensor requesting it for internal stuff, or if it's the FPGA doing it and maybe the FPGA is doing some crude internal auto exposure or maybe just sanity checks. The shutter can be controlled via can bus commands per Tmbinc's documentation but I haven't figured out anything other than the UDS challenge.

[tmbinc]

For shutter operation, you need to send on can_id 0x401 the following data:

"c8 64 18 00 04 00 00 00"
"c8 64 18 00 00 00 00 00"

Byte 7 defines what's shown during the shutter operation (black, uncorrected, test pattern).

[Treehouseman]

Do you have a list of what commands you saw in the microblaze code? Or at least where the block of code is that was processing the messages? I never actually found that, I got the key from simply finding the 16 bytes in an out of place location and seeing that there was a ref to it. I was just doing trial and error with high entrophy 16 byte blocks.

[tmbinc]

The key is in the boot block (initialized BRAM from the bitstream); the CAN handlers are in the firmware.

This is what I have there: (All names from me of course)

ROM:2200E970 00 00 04 01     CanFuncTable:   .int 0x401              ; DATA XREF: _InitCan:loc_22001B6C↑o
ROM:2200E974 22 00 1A 10                     .int CanCamFuncControl
ROM:2200E978 00 00 03 04                     .int 0x304
ROM:2200E97C 22 00 19 20                     .int CanCamSync
ROM:2200E980 80 00 03 13     CanFuncTable2a: .int 0x80000313         ; DATA XREF: _InitCan+6C↑o
ROM:2200E984 22 00 04 88                     .int CanCamStatus
ROM:2200E988 00 00 03 11                     .int 0x311
ROM:2200E98C 22 00 05 FC                     .int CanCamSerialNumber
ROM:2200E990 00 00 03 12                     .int 0x312
ROM:2200E994 22 00 05 70                     .int CanCamInitRequest
ROM:2200E998 80 00 03 03     CanFuncTable2:  .int 0x80000303         ; DATA XREF: _InitCan+78↑o
ROM:2200E99C 22 00 04 88                     .int CanCamStatus
ROM:2200E9A0 00 00 03 01                     .int 0x301
ROM:2200E9A4 22 00 05 FC                     .int CanCamSerialNumber
ROM:2200E9A8 00 00 03 02                     .int 0x302
ROM:2200E9AC 22 00 05 70                     .int CanCamInitRequest


So, 0x401 is handled in CanCamFuncControl (22001A10).

[oddbondboris]

Got a bit more progress, I'm really starting to fight signal loss, I'm going to have to rework the connection between the deserializer and the camera, I'm not sure what makes the difference between a good capture and a bad one, without touching the camera I can get a good 30 second capture and then instantly running a second one get maybe one or two good frames out of the entire thing.

if you have a cheap fx2 based logic analyzer (or any streaming output logic analyzer) you might wanna hook it up to the deserializer output and see if what the ftdi chip is spitting out matches.

[Musclor]

This is pure genius and if related to the old thread where someone had asked if such thing was even possible, even better. Absolutely inspiring! Bravo!

[oPossum]

Using dsPIC33EV256GM102
<click> <click>
Thanks to tmbinc, Treehouseman, Lennie, and Johan for their help with this.

[oPossum]

Using the ubiquitous ELM327 cable

[conmega]

Well this is the best I have gotten out of it so far.
Still some noise issues due to me just hooking this straight to the FPGA using its internal termination.

Thanks to oPossum and Treehouseman for starting me down this rabbit hole.
Also thanks to tmbinc. I took the Migen implemented de-serializer and  decoder and was able to get it ported over to 7-Series parts (mostly changing the SERDES units over to the new ones and dealing with different clocking resources).
I took just the fpdrecv.py and basically built it out as top then implemented some clock domain crossing/fifo and LCD controlling in VHDL and glued the two together.
Its been a heck a week working on this.

[ArsenioDev]

Conmega, that is one IMPRESSIVE quick build you have there. Only wish I could poke around in it, been thinking of how to port it to the ECP5 boards I have, should be enough LUTs to drive a smaller display and do the CANBUS on board.

[conmega]

Thanks, but most of the heavy lifting was done by others, especially the decoder/deserializing written by tmbinc.
I simply had to learn a bit of Migen to get it move from a Spartan-6 target to 7-Series parts.

My design as it currently sits uses around 700~ LUTs, 1100~ FlipFlops, and two half BRAM units, which are 36x1024 and can be used as 18x1024 "half" BRAMs.
For reference this is only about 2%~ of the entire Artix-7 100T... So I bet if you had the bandwidth or implemented an even lower sample rate over-sampler you could probably get away with running this on something much smaller and simpler like an ICE40, I will probably plan to just target a small Artix-7 or Spartan-7 myself.

The big thing is to use tmbinc's decoder/deserializer directly without modification. You need at-least a 1080Mbps capable SERDES. Looking at Artix-7 and Spartan-7 it looks like speed grade 1 always is 950Mbps capable and speed grade 2/3 is 1250Mbps.

Of-course this is assuming you don't want to modify tmbinc's sampler code which over-samples by 7.7x as described on his website.
You could probably modify it to work with 6x oversampling (840~Mbps) to allow it to work on more common/cheaper/smaller parts.
But its easiest to just use a speed grade two 7-series part. I had to change the clocking.py to implement 7-series primitives for the MMCM and clock buffers. And the Serdes change was a bit more complex but not really all that bad. To get 1080Mbps you use the ISERDES2 in "OVERSAMPLE" mode, which has a 4 bit output but allows for QDR sampling. So you feed a 2 times data rate clock, in this case to match tmbinc's code which expects a 7.7x sampling it would be a (135Mhz*2) 270Mhz clock then you get 4 bits every 270Mhz which you put into an 8-bit register every 135Mhz and you get your 8-bit 7.7x sample data to feed into the sample.

So its not that bad, honestly it took me a few days to learn enough Migen to get stuff built and then a few days to understand the timing coming out of the sampler and to just write sane VHDL... I am still quite new to FPGAs so there was a good 4-5 days of buggy glitchy video on the screen

I will probably try and get his sampler down to only a 6x oversample rate so I can use a common and cheap FPGA board (like the Cmod S7) with a speed grade 1 part on it for implementing this into a hand-held device.

[ArsenioDev]

The good news is even the basic ECP5 Serdes are 3.2Gb/s! Hmm, guess it's time to nose to the grindstone on the Migen and Litex stuff eh? Honestly I'm trying to get this all running on FOSS flows, gonna do the designs for a custom breakout of the FPDlink. Prob gonna make a few and give em to friends then open source the design.

[conmega]

And now glitch free video with hardware histogram equalization!

This uses the current frame to build a histogram in BRAM1 then between frames using downtime to built up an equalized histogram in BRAM2 (in this case equalized to 6-bits for the 6-bits of green available).
Then using the new equalized histogram in BRAM2 we apply that to the next frame's pixel data on the fly while building up a new histogram for the next frame in BRAM1 again. This is while not 100% accurate from frame to frame its close enough and extremely low latency. It is also space efficient requiring about 1/3rd the amount of BRAM to store both histograms as it would take to store one entire frame. Not to mention the logic to say offload this to a microprocessor for processing via software

I also adjusted the focal point of the camera by unscrewing the lens outward a few turns as Treehouseman found will bring the focal distance in.
I was about 2ft~ away from the camera and I had gotten it to about 18~in or so before!

[conmega]

And now with palette support!
Thanks to oPossum for generating a few palettes for me to test out.

Also decided to show off the histogram equalization a bit with the Metcal iron

[dnhkng]

I'm not asking for a key, but is there a dump of the flash uploaded somewhere? It seems silly that we all have to desolder the SPI chip over and over again.

And thanks to all the posters on this thread so far, awesome work!

EDIT:
Note: I don't have an NV2, so I'm doing this without feedback.

Not made much progress yet, but I will post a (growing) list of things that DON'T seem to work 

I *assume* this is how the unlock protocol works: the camera sends you a 'Seed', as per Tmbinc's blogpost description, and this data is 12-bytes random plus 4-byte serial number. You have to use a secret 'key' and do an HMAC-MD5 digest of the 'Seed' and send it back. The camera has done the same thing and compares its calculation with yours, and if they are the same, it unlocks itself. No secret key, no unlock. It seems simple enough...

I tried using the screenshots from oPossums post, where he showed 2 seed/key pairs, 16 bytes each.

Code: [Select]

seed:    8D E9 AD 49 DC 8E 68 9E CE F6 89 20 3C 25 DB 53
Key:      90 CC 53 17 64 A1 66 1C 58 0A 88 16 EB 39 50 52

Code: [Select]

seed:    7E 71 63 31 D5 5E EF 26 A1 9C AA A1 3C 25 DB 53
 Key:      BA CC 53 C6 8A 79 CF C2 46 AD 56 AE C3 B6 A4 68
Then, because I'm dumb, I just brute-forced iterated through the whole BIN file, reading every sequential 16-byte block as a potential private key.
I used the standard python hmac library.

Code: [Select]

import hmac
output = hmac.new(password, seed).hexdigest()
then I tried to match the posted 'Key' with the calculated output, and the whole process of 8 million MD5-HMAC digests (the bin file is about 8 Mb) takes under a minute, but no luck finding the key    I might try iterating backward next, because why not?

Just in case there is data in a different format, I did the same again for the Data2mem output file, after extracting out the binary data. That didn't work either.

I was a bit confused about the BRAM columns/rows memory layout (I have used a MicroBlaze core before, but I just drew in wires from the Microblaze to the BRAM controllers in Vivado, I have no idea how they work under-the-hood), but I tried to take the 4 rows that kind of looked interesting and interleaved them together in various ways. From the discussion here, it seems that they have to be interleaved somehow (maybe the MicroBlaze reads across down rows?). The most interesting 4 rows have most of the data at the beginning, so maybe stacking the rows and then reading in columns makes sense...  Eyeballing they merged HEX data it didn't look like much beyond looking at a The Matrix screensaver. I didn't find a nice, simple, stand-alone MicroBlaze disassembler, plus I'm too lazy to learn IDA for a single project 

EDIT:

I found the key last night using the approach above, and the tip from tmbinc and oPossum below (thanks guys!). Just by knowing the XOR ordering (x4, as you don't know the byte offset), and overlaying that with the permutations on interleaving the 4x BRAM rows (x24), running over every possible private key in the data2mem dump (2Mb) shouldn't take more than half an hour brute force (~200 million potential keys). This should be faster than installing all the software you need to go the disassembler route. Using the extra tip from oPossum makes the calculation a bit faster. The timing below including passing the Data2Mem file for BRAM data too

Code: [Select]

34.2 ms ± 453 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

[tmbinc]

The seed/key pairs are correct. Your code should be correct as well.

However just iterating over the flash file will not work because it only stores the XOR value (55AA33CC) from the data in BRAM. If you xor them together, and put them in the right order, it should produce an 16-byte ASCII string with two names.

[tmbinc]

By the way, if someone wants to post the key, I'm fine with that, but _I_ will not be that person .

[oPossum]

Some hints...

Row 06 XOR CC
Row 09 XOR 33
Row 07 XOR AA
Row 04 XOR 55

Reply #33 (not the screen cap)

[railrun]

So many hints, but still no approach. 
@dnhkng Thanks for your mail. Unfortunately your mailbox is full and I couldn't answer your last mail.

[Fraser]

I have one of these cameras and it’s associated original controller languishing in my “may be useful someday” cupboard 

This is a very interesting thread but I am no coder and much of it is beyond me. Maybe one day someone will offer an adapter board and code to release the Autoliv camera from its shackles. If that day comes, I will be a customer for the solution. Until then, the Autoliv camera remains an interesting paperweight in my collection 

To the knowledgeable participants in this development work.... I salute you 

Fraser

[Treehouseman]

I have one of these cameras and it’s associated original controller languishing in my “may be useful someday” cupboard 

Well for the NV2 there's not much need for the controller at this point

Now the NV3, we have a camera, we've started looking at it and tried to figure out how to pull the softcore out of the flash, but it's proving to be quite a task. A controller for that might prove to be helpful, or at least a capture of the comms from when it unlocks to see what commands it's using.

[conmega]

I have one of these cameras and it’s associated original controller languishing in my “may be useful someday” cupboard 

This is a very interesting thread but I am no coder and much of it is beyond me. Maybe one day someone will offer an adapter board and code to release the Autoliv camera from its shackles. If that day comes, I will be a customer for the solution. Until then, the Autoliv camera remains an interesting paperweight in my collection 

To the knowledgeable participants in this development work.... I salute you 

Fraser

I have been developing a little board that allows one to hack an RJ-45 onto an Autoliv2 and then my board has an S7-Mini FPGA board, RJ-45 for the camera allowing one to use any standard ethernet cable, and a USB-PD spoof board allowing one to power the entire thing off a USB-C PD capable power bank or supply. It also has a 320x480 LCD on it to allow for viewing of the full resolution.

Now I have been working on this board but have been hesitant to want to sell them to others for the same reason tmbinc is hesitant to release a key...
I mean I could sell it without a key but entering a key over a 4-button keypad or having to rebuild a bitstream to throw it in both seem like silly things. So likely I will just be keeping this between me and a few people who have been hacking on this with me.
But you can see what I have made here:
https://twitter.com/ConnorKrukosky/status/1271046037742989313

I could also sell the boards with them artificially locked to 9-fps with some challenge to get 30-fps...
Probably something more complex than a konami code or something though

The big challenge ahead on the NV2 is getting temp data correlated to what comes out of the camera... That is where I am currently procrastinating... Otherwise I have a RiscV softcore doing the unlock as seen in that tweet and I have all the palette application and histogram done in hardware. I also have the RiscV able to load other palettes off SPI.

Honestly though it would be smartest to design a board that had everything integrated and not using an expensive FPGA dev board on it... Its an overkill FPGA for the application too.

But I was just focused on getting something I can have one of and use as a tool. Not something I planned on selling.

[tmbinc]

Really glad to see all this progress here!

The two elephants in the room are "signal cleanup" and "temperature extraction".

For the "signal cleanup", it would be interesting for example to look at a Flir E40 (which uses the same sensor), and see which steps they do. The NV2 has a lot of "structured" noise that I think the Flir E40 is able to clean up (not sure if at the expense of sensitivity).

For the "temperature extraction", I'm pretty sure we have all the data we need - there's calibration data stored on the device that can be read via UDS, and there's a number of temperature sensors that is transmitted via CAN and via LVDS.

The NV2 does automatic FPC, however it's not very good at it, there's a significant "remain" of previous image data. I can't judge why really. I've tried to poke FPGA registers to receive the raw sensor data but couldn't do it. It would be great if we could figure out what corrections the FPGA actually does. We can grab the intermediate NUC pictures from either flash or RAM, and see how they compare to the sensor calibration data stored in Flir Exx cameras. I suspect the algorithm to be identical or very similar - likely it's just an offset and a scale?

Let's turn the NV2 into a top-notch thermal camera!