OpenIRV. ISC0901B0 (Autoliv NV3, FLIR E4/5/6/8) based opensource thermal camera

[VGN]

What OS are you using for the firmware?  You might look into the Zephyr project.  It is an embedded RTOS supported by the Linux Foundation.

Hi! Oh...forgot to say, I'm using FreeRTOS. It is running on 100MHz 32-bit Microblaze MCU, avaliable RAM size is about 8MB.

First, congratulations for your work, this is simply awesome !!
REMOVED : I didn't see you asked for C gui, not C++

Hi, thanks! Yes, unfortunatelly ImGui, that you have recommended is C++, though there is a cimgui project, that wrappes original ImGui API: https://github.com/cimgui/cimgui

[pansku]

Looks like a very interesting project and I'm very impressed by the professional level of the design. Bookmarked for sure and seems like I might have to get one of those NV3 modules before summer 

[swwils]

Did you consider helical barrel arrangement for focus instead of the screw? I think a focusing helicoid would be doable with small nylon washers.



This massive one uses actual bearings but you get the idea.

[VGN]

Looks like a very interesting project and I'm very impressed by the professional level of the design. Bookmarked for sure and seems like I might have to get one of those NV3 modules before summer 

Hi, thanks! Consider email me if would like to become a beta-tester (details are in the first post of this thread).

Did you consider helical barrel arrangement for focus instead of the screw? I think a focusing helicoid would be doable with small nylon washers.
This massive one uses actual bearings but you get the idea.

Hi! In fact, no, I haven't considered this way of focusing. I'm not really good with optics, but I think that helical focusing mechanics are mostly needed for zooming, where large lens travel is required. But this camera isn't supposed to have zoom (except digital one). The main idea was to overcome the fixed lens focus distance range limitations (factory focus distance starts from ~3 meters). Fortunately 2mm lens travel screw-based focus is enough to cover distances from ~8cm to infinity. It is also quite cheap and reliable (almost no wear), reused quite a lot of original camera parts. Moreover I have redesigned some parts and fixed some problems with speed, vibrations and noise (will make a post later).

At the same time I understand that somebody probably have much better IR lens. And I don't see any problem to integrate any non-standard IR lens with this camera. Some custom plastic adapter is just needed (FDM printer capabilities should be enough). If you need some more information (dimentions or etc...) to design an adapter, just let me know, probably I could help.
This is the power of opensource.

[Vipitis]

As for lens mount, that is really a difficult one. As I am planning my own thermal camera project, it includes custom housing and a lens mount that supports interchangeable lenses. So no screws or sealing. Just swapping lenses in the field with rear lens caps and mount caps.

I have been thinking about something that locks securely like a PL mount, but the flange distance of what I can work with don't allow that at all. As one of the lens I am planing to use is rather large. It will have to be mounted to rails or a plate. Which makes changing lenses by turning and such more difficult.

I have seen screw mounts in a lot of product, mostly M39 and some smaller stuff among the ThermApp and Thermal Expert dongles. While those sensors are about the same size as the one you are working with, that's not the case for me (I am looking at 21mm diagonal - means I might be able to use a lot of older lenses and get them from sources like Bill). FLIR had their own mount for the higher end cameras like Txxx and Ex5. But it's a mount with electronical connecting for auto focus and saving a lens correction profile.

I am looking forward to what you come up with and might see a way to adapt that for my own plans... Or not.

[swwils]

Ah I understand, great!

I have bought the camera module (AUDI) and am ready to fabricate PCB.

Do you have a current BOM or should I take from schematic?

I plan to make a simple Ge or ZnSe plano-convex lens to use as macro for this system. Like described here:

http://uvirimaging.com/wp-content/uploads/2018/03/Diy-Thermal-Camera-Telephoto-Converter.pdf

This combined with the ready made optics housings from thorlabs (Ø1" Optics) - will make a very very decent and cost effective tool!

[VGN]

I have been thinking about something that locks securely like a PL mount, but the flange distance of what I can work with don't allow that at all. As one of the lens I am planing to use is rather large. It will have to be mounted to rails or a plate. Which makes changing lenses by turning and such more difficult.

Why not using a special flange adapter to get the proper flange distance for your large lens?

I have bought the camera module (AUDI) and am ready to fabricate PCB.
Do you have a current BOM or should I take from schematic?

I haven't finished testing hardware yet, hope to finish by the end of this month. Also this is supposed to be a kit with fully assembled PCBs + enclosure and other mechanic parts, though for the most brave and experienced of you, I could ship bare PCBs with BOMs, assembling schemes, etc...

[Spirit532]

Yesterday I came across a LVGL library: https://lvgl.io/, github: https://github.com/lvgl/lvgl
LVGL looks promising, worth to try. Any thoughts?

I highly recommend you go with LVGL. It's extremely simple to implement, can be augmented with 2D acceleration, and is fairly easily extensible.
I'm using it in a commercial project right now, no issues whatsoever.

[VGN]

I highly recommend you go with LVGL. It's extremely simple to implement, can be augmented with 2D acceleration, and is fairly easily extensible.
I'm using it in a commercial project right now, no issues whatsoever.

Hi, Spirit532! Thanks, going to try it!


Meanwhile...updates!

Ok, looks like I have DVI/HDMI working. For now three screen modes are supported: 800x600, 720p and 1080p.
I was a bit surprised with stable operation of the 1080p mode, as it is quite far away from FPGA capabilities according to specifications.
Anyway, it works perfectly over a 3 meter cable:

[ArsenioDev]

WHOA, that is AMAZING. Cannot wait to add this to my lab debug toolkits as an overhead capture camera, will help A TON, can just dump output into my many HDMI capture inputs.

[SilverSolder]

Truly impressive!

What is a good HDMI capture device?  The ones I have tried in the past all end up over-compressing the image...

[smason]

If youre plugging into a computer, you can look into an Elgato Camlink. Using OBS you could record quite well. this hard worked for me in the past.

[oPossum]

From the first post...

Supported outputs:
1. 320x240 TFT LCD
2. USB UVC (a common webcam protocol)
3. HDMI 480p/720p (probably even 1080p with Spartan-7 FPGA)

[fest]

Received an Audi 4G0980552 part. Although it is missing front window, I was assured by the seller that it's working. By pointing out that without the window, it's worthless for it's intended use, I got it for 160EUR.

Inside is surprisingly intact- no signs of water ingress. I was surprised to find a desiccant bag inside the camera. Judging by the oxidation layer that fell off from the main enclosure screws, nobody had opened it for a very long time, so it appears to have been there by design.

It does appear to be NV3 by the markings on the PCBs.

[VGN]

Received an Audi 4G0980552 part.

Hi, fest! Thanks much for sharing photos!
You have finally approved that Audi's 4G0980552 is exactly NV3.
BTW, internal condition is indeed perfect.

[fest]

Before trying to interface the sensor, I wanted to better understand the signals. I traced required signals on sensor board based on OpenIRV's M-board pinout- see attachments. Unfortunately, the NV3 core board does not have nice testpoints for CMD and BIAS signals, but I found them accessible on resistors on the other side of the board.

VGN, do you know if FPGA queries the sensor at all, before receiving a key?
Just probed the DATA_EVEN and CLK lines- there is a ~75MHz clock and some activity on data line (at least, as much as I could see with DS1052). Judging from the captures in Vue336 interface thread, DATA_EVEN comes from sensor only, so it appears the FPGA is receiving data from sensor even without any security key.

[VGN]

VGN, do you know if FPGA queries the sensor at all, before receiving a key?
Just probed the DATA_EVEN and CLK lines- there is a ~75MHz clock and some activity on data line (at least, as much as I could see with DS1052). Judging from the captures in Vue336 interface thread, DATA_EVEN comes from sensor only, so it appears the FPGA is receiving data from sensor even without any security key.

Exact clock value is 73.636MHz. There is no any encryption or any other secure communication between the camera and FPGA. If you send clock, valid command and valid bias values, you will get valid data on both data lines (even + odd).

[fest]

VGN, you mentioned the difficulty of obtaining bias values for each pixel. Looking at your HDL sources, I now understand that a bias value is sent by FPGA to sensor, for each pixel? That is quite interesting- in that case, it seems to me that there is a significant manufacturing variation for each microbolometer (that is, even more significant than the usual non-uniformity which is handled by the NUC).

Can you share the bias values for your sensor? I wonder if there is any discernible pattern in the values.

Anyway, I'm especially curious about this, because I hooked my unit to a logic analyzer, and the FPGA doesn't seem to send any bias values at all. Consequently after the per-frame command packet, I see something that resembles one image line, and then only the preamble on the data lines from sensor. I wonder if it's a security feature (I've heard about lockdown mode if FPGA receives incorrect data on NV2), or my unit is somehow damaged. There does appear to be some activity on the SPI flash lines, but I'll probably try dumping it next.

Failing that, I wonder what is the way detectors are characterized at the factory. Maybe it's just as simple as as binary search on the bias value for each pixel that yields the most uniform output of the detector.

[VGN]

VGN, you mentioned the difficulty of obtaining bias values for each pixel. Looking at your HDL sources, I now understand that a bias value is sent by FPGA to sensor, for each pixel?

Yes, that is right. FPGA sends 6-bit bias value for each pixel. (In fact it sends 7-bit, but only 6-bit are significiant).

That is quite interesting- in that case, it seems to me that there is a significant manufacturing variation for each microbolometer (that is, even more significant than the usual non-uniformity which is handled by the NUC).

This is also true. There are two ways to solve this problem. Bias data and command word can be retrieved from original board NOR flash (I have some success here). Another way - to recalculate this values over a two-point calibration.

Can you share the bias values for your sensor? I wonder if there is any discernible pattern in the values.

Follow this reply: https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/msg3254678/#msg3254678
I used fixed 0x25-0x26 bias value for all pixels.

Anyway, I'm especially curious about this, because I hooked my unit to a logic analyzer, and the FPGA doesn't seem to send any bias values at all. Consequently after the per-frame command packet, I see something that resembles one image line, and then only the preamble on the data lines from sensor. I wonder if it's a security feature (I've heard about lockdown mode if FPGA receives incorrect data on NV2), or my unit is somehow damaged. There does appear to be some activity on the SPI flash lines, but I'll probably try dumping it next.

This is ok, camera do not output any bias values at this point, it waits for some command from the control unit. Don't worry, your unit is not damaged, as you see preambles at the data lines (10101010101010). Preambles are generated by sensor, that means it is alive. As I said previously, there is no any secure protocols between sensor and FPGA. First three lines - are a pipeline garbage, it doesn't look like to be a thermal data, no correlation with scence or sensor temperature. The first image line you will get at the 4-th line output. But now you see zeros after preambles 10101010101010, as you do not send any bias at all. Also there is a huge pipeline delay (3 lines) between transfer of the bias values and pixel output. After command transfer we start sending bias values, then we will see 3 strange data lines from sensor (I call it pipeline garbage), only after that we will get valid thermal data. Checkout ISC0901_capture.v design file.

Failing that, I wonder what is the way detectors are characterized at the factory. Maybe it's just as simple as as binary search on the bias value for each pixel that yields the most uniform output of the detector.

I think you are right. There is only 64 levels of bias. We just have to take 128 pictures from two different simple blackdoby sources, find the bias values for each pixel that gives the best pixel dynamic output range. But picture will not be uniform at this step. We will have to calculate individual pixel gain value to get uniform image.
In fact I have already tried to calculate pixel gain values for fixed 0x25 bias. Quality of all videos and photos from my thermal camera in this thread is based on my own custom calibration.

[VGN]

Hello, some updates...

Finally have USB UVC working. Tested on linux, windows and even android. Sorry, no video for this time, but a photo.
I haven't pushed new commits for a long time, going to fix that in a few days. By the end of this week I'm going to finish AV (analog video), this is the last hardware subsystem, that I must check.

Also I have started working at documentation about the sensor, including pinout, waveforms, command pattern (at least the things I know and understand right now). This document will be updated in case of any new details.

[bap2703]

I think you're doing too well by yourself for people to post useful comments.
So in the end I can just say you're doing an amazing work 

[Hydron]

I agree that there is some fantastic work going on. I only wish I had more time available to look into your design in more detail - I would like to have a go at interfacing a ULIS sensor to it in place of the FLIR one, as I have a couple without working cores attached (both VGA and QVGA).

[fest]

Which ones do you have? Most of them seem to have analog output, although some of the newer ones have digital output (which, although is simpler protocol wise, is not compatible with ISC0901B0).

[VGN]

I think you're doing too well by yourself for people to post useful comments.
So in the end I can just say you're doing an amazing work 

Thanks for your support! It really helps to keep working)

I agree that there is some fantastic work going on. I only wish I had more time available to look into your design in more detail - I would like to have a go at interfacing a ULIS sensor to it in place of the FLIR one, as I have a couple without working cores attached (both VGA and QVGA).

Thanks! I think this is possible to do, but you will have to make a special adapter board for your sensors, check and tune power rails to suit ULIS power requirements and implement a HDL design for FPGA that will capture raw data and feed to the image processing pipelines. I keep in mind, while developing, that image processing cores should be able to support other sensors and resolutions.

Which ones do you have? Most of them seem to have analog output, although some of the newer ones have digital output (which, although is simpler protocol wise, is not compatible with ISC0901B0).

For this case we have 20 high performance IOs at the sensor connector of the M-board. I think this is enough to connect ADC, which can be placed on the special adapter board. Yep, this is quite tricky, but worth doing if you want to bring alive some really cool and expensive sensor.

[Hydron]

Which ones do you have? Most of them seem to have analog output, although some of the newer ones have digital output (which, although is simpler protocol wise, is not compatible with ISC0901B0).

VGA sensor has analogue output only, QVGA has digital as well (with a different package/pinout though ). Would only be using analogue option, and I have the power supply, biasing and ADC in hand, it's the HDL design and modification that would be the big job.