Autoliv NV2 on Raspberry Pi

[dnhkng]

I'd like to try working on the image clean up. That more my thing. Conmega, the S7-mini board looks great, I'll order one from Trenz. It seems like a fair price to avoid having to lay out and solder a chip like that.

I think I might have the same parallel LCD from your first video in my parts-box too. Can you access image data easily over USB with your S7-mini setup? I'll need lots of data to play with for image correction. Maybe a linear fit in enough, but from the temperature sensors I have used in the past, it's been a more complex model.

To get a good cleanup, it might be good to really defocus the lens to build a pixel-correction array.  Defocused imaging a piece of melting ice and boiling water gets you 2 fixed points to work with.

EDIT:
I've written micropython code for the unlock (HMAC-MD5/MPC2515 driver), which should be great on FPGA using FuPy https://fupy.github.io/. With a serial interface into the REPL, you can mess around on the CAN bus directly (no messy compile/upload steps).  I'm using as MCP2515 board with SPI, because they are cheap and easy. Maybe a CAN bus IP on the FPGA connected to a transceiver IC (MCP2551), and hooked up to FuPy on a RiscV core is a nicer way to go.

Before and after 'clickclick' traces 

[dnhkng]

Kinda got some pictures out of the camera!

well not really...

I hacked together a hand-soldered monstrosity, wired up to a fake Saleae Logic thingy, to capture the upper 5 bits of thermal data, and the h/v-sync and clock lines, then chucked it all into a numpy array to poke around. I haven't figured out the logic of the video signal yet, so I just grab every line I can that has 326 pixels between syncs, and I get something that looks like my hand holding a lighter over and over (I captured 300 mS of data, so 10ish or so frames I hope).

It looks like either blocks of rows are scrambled or just missing entirely, kinda weird...

@Treehouseman - Is this the same issue you had?

[Treehouseman]

I'd say same symptom and same general fault but likely different cause. I was suffering capture rate issues, with things being pushed to the limit and resulting in temp fluxuations in the camera causing me to lose sync. For your issue your'e not consistently capturing data it seems, looks like you're getting bursts of valid data. I know we were looking at the FX2 (aka your clone) for possible data dumping, but it would have required custom firmware for all the bits,

[tmbinc]

There are empty lines between the valid lines, which you need to filter out. I think this is done to keep the output data rate constant, so it doesn't need to be locked with the sensor scan rate.

[dnhkng]

Got it!  It seems the ground wires are in fact useful... 
@ Treehousea and tmbinc - Thanks for the tip! I thought the data was sent bursty, but it seems if you hook up the ground wires its a continuous stream! 

Still a bit confused about the frame sync bit, I seem to get more than I would expect (bit 15 goes high 4 times per frame, but its a regular pattern), but I'll work it out, it's a fun puzzle.

Pretty happy so far, all done in Python, for less than $20 in bits and pieces. Haven't had to dust off my Arty A7 yet!

[oPossum]

Bit 15 is high for the first line and line number 20 (not the 20th line). So twice per frame.

[dmendesf]

Am I crazy or the homebrew NV3 thread just vanished?

[oPossum]

https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/

[dmendesf]

Thank you and sorry for the panic attack

[kmike]

Hello,

if I understand correctly, the bits 16 to 23 are not used after deserialisation? I plan to use the DS90C124 for the task and would like to save some FPGA pins 

Br,
mike

[oPossum]

Bit 14 and 17 to 23 can be ignored.

0 to 13 contain video or line/frame number
15 and 16 are sync

[kmike]

thanks!

just for my planning: how much current does the camera need on the 12V rail, when it is operating and unlocked?

Br,
miike

[oPossum]

Deserializer and MCU board. Currently connected to a QMTech Cyclone 10 dev board.




First light

[dmendesf]

Deserializer and MCU board. Currently connected to a QMTech Cyclone 10 dev board.

Congratulations. When will you launch it on Tindie?

[oPossum]

Low frame rate (~5 fps) video via serial port.

[kmike]

would someone be so kind and send me a dump of the spi flash?

My camera looks pretty new, and I dont want to disassemble it

Thanks in advance!

br,
mike

[oPossum]

Full frame rate (30 fps) and full resolution (324 x 256) using an FTDI FT2232H in 245 async mode.
Video shows palette selection, dynamic palette (DP), and histogram equalization (HEQ).

[Imolteor]

Hi, I realise I'm a bit late to this party. I'm also brand new around here (please excuse me if I have muddled any etiquette rules). I'm a postgraduate at a university in the UK looking to use thermal imaging for wildlife conservation.

We can't afford much (given life) and this sensor is awesome, plus the work thats been done.

I'm going through the thread looking to piece together what I need but I'd really appreciate a summary to getting started.

If I can get one of these off the ground on my end I'll be pumping a lot of work into using it. Any software I write to use it I'll make sure to post here for you guys as well, in case you find it useful.

[dnhkng]

You need just need 2 things:

• A way to unlock the camera
• Some fast hardware to get data off the camera module (FPGA)

The first is super hard to figure out, but it's been solved, and there are hints throughout the thread.
The second is an open question. Some people have posted demo's, but I don't think there is public FPGA code out yet?

Best of luck!

EDIT: I have some rough FPGA code available soon:

If you cheat a bit, you can use the gap between the valid lines as the H-Sync, and the gap between frames as the V-Sync.
There are sequential lines with no gaps (about every 7 lines), so you lose those lines (there's no time for an H-sync pulse). This compresses the image vertically, but as the position of those double line jumps around, the image looks pretty stable overall.

But, what you get in return is super low resource requirements:
- No BRAM or SDRAM used at all (we don't need a frame buffer)
- only 58 LUTS and 21 registers used in total for the most basic implementation (as in the image)
- about 300 LUTS using histogram shifting and a colourmap (https://ai.googleblog.com/2019/08/turbo-improved-rainbow-colormap-for.html)
- you can drive this directly from the recovered clock on the DS90C124 (CPLD without a crystal?)
- probably drive a 320*240 RGB really nicely!

As its a PITA (at least for n00bs like me) to get a softcore on an FPGA, and ESP32's are super cheap, I will use that for the unlock, and try and get it to act as a WIFI video streamer

[tmbinc]

Nice!

By the way, I have demo code on https://github.com/tmbinc/fpdlink but it uses an OpenVizsla so it's not exactly the most practical setup. It doesn't need the DS90C124, but uses the FPGA LVDS receiver. There has been some other work to bridge this to a RasPi over SPI but getting SPI to work at a high performance is not trivial.

[dnhkng]

As perfect is the enemy of good, here's some (very imperfect/not very good) minimal HDL code to at least generate an image on an LCD at 30FPS, using the DS90C124 decoder IC:
https://github.com/dnhkng/NVCAM-readout

You need to just wire the DS90C124 up to the cheapest FPGA you can find that has enough pins. Using the deserializer with a tiny FPGA (< 1k LUTs)  is at least hand-solderable stuff, and won't cost more than a few dollars more than an FPGA with SERDES. I used this cheap dev board:
https://www.seeedstudio.com/Sipeed-TANG-PriMER-FPGA-Development-Board-p-2881.html

and used this screen:
https://www.seeedstudio.com/5-Inch-Display-for-Sipeed-Tang-Nanno-p-4301.html

because it's cheap and they fit together.

This particular screen does NOT give a shit about regular timing signals/clock speeds. I have a hard time giving it a signal that doesn't cause something reasonable to appear on the screen. Due to the timing stuff I mentioned in the post above, you get some extra weirdness on the right-hand side of the screen, due to the internal line buffer in the LCD controller hardware.
There are a few demo projects for that FPGA, including a demo for the LCD. Use the IO config file from that project for the LCD IO's :
https://github.com/Lichee-Pi/Tang_FPGA_Examples

The FPGA has 20K LUTs and 64MBit SDRAM etc, so there is lots of room to improve things, like add a frame buffer, or colour-maps, or SPI stuff to raspberry pi, or even add in a RISC-V core for the unlock.

You'll have to perform the camera unlock separately of course.

[dnhkng]

I've added some more stuff:



- I now use a framebuffer
- Use multiple colourmaps
- Do either histogram normalisation (not equalisation, to keep the values linear), or show the lowest 8-bits for maximum contrast
- Plot the histogram of the un-normalised data, so once we figure out the calibration, we have a histogram of true temperatures.

This is currently all done on FPGA without a RISC-V core, because I can't figure out how to use one... I should also learn how to write test benches...  And how to structure VHDL...




ToDo:
- figure out the calibration
- add an optional crosshair, and display the centre temperature on the histogram plot
- write an SPI controller, and interface with an ESP32
- write the video streamer on the ESP32 web server

[smason]

Great work so far! I purchased one of these cameras in hope i will be able to buy a kit in the near future to retrofit this into my car!

I will keep an eye on this thread for your progress!

[twizzter]

Hello from yet-another-scrapyard-nv2-owner!
Got flash dump, tried to perform an initial analysis and i stumbled onto weird cpu_rec output:

Code: [Select]

mint@mint:~/Desktop/nv2$ python3 cpu_rec.py nv2.bin
nv2.BIN
full(0x800000) None                               chunk(0x1b800;55)   6502      which seems to be invalid, as i haven't expected 6502 code here.
 
binwalk shows:

Code: [Select]

mint@mint:~/Desktop$ binwalk nv2.bin

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Xilinx Virtex/Spartan FPGA bitstream dummy + sync word
54420         0xD494          Intel x86 or x64 microcode, sig 0x04a00004, pf_mask 0x822000, 20C0-01-10, rev 0x-3ffefdf0, size 657408
1984064       0x1E4640        MySQL MISAM index file Version 4
3210374       0x30FC86        MySQL ISAM index file Version 8
5051898       0x4D15FA        MySQL ISAM compressed data file Version 6
Is it possible this dump could be encrypted? Or maybe i forgot to add some command line flags when using cpu_rec?
 

                                                                   

[Wazowski]

Is there a kit available for someone with zero coding experience. I would love to use this device in my car. It has huge potential for good. Excellent job so far but it has to be dumbed down for the rest of us.