OpenIRV. ISC0901B0 (Autoliv NV3, FLIR E4/5/6/8) based opensource thermal camera

[VGN]

Hello everyone!

This topic is about my attempt to develop an opensource high performance and multifunctional thermal camera based on a ISC0901 sensor.

I use thermal imaging to diagnostic malfunctions of different devices under development or repair. Five years ago I have developed a small camera based on FLIR Lepton 2 80x60. It works pretty well, but in fact this is a toy for me, and I was looking for something better. Unfortunately FLIR EX series didn't suit for me too for many reasons, including poor fixed focus lens, a bit inconvenient form factor, price, etc...
I'm also a supporter of opensource projects, but all of them are still based on FLIR Lepton 2/3 products or some other tiny sensors.


Once an Avtoliv NV3 camera module occured in my hands. I've disassembled it and found out that this camera module has a great potential and perfectly suits my new project, because:
1. It is based on very popular ISC0901 336x256 sensor. Popular, because NV3 camera is used on tons of different vehicles till now, you can find this spare part very easily. FLIR EX and ETS320 are based on the same sensor.
2. It is an external shutter based camera. External shutter allows to significially improve the image quality.
3. It has a pretty fast (15mm in dia) telescopic lens! This lens is much better that FLIR EX lens.
4. Pretty nice form factor (60x62mm)


I decided to develop a camera that will be multifuntional and can be used in many different ways and applications:
1. Hand tool, like FLIR EX
2. Stand tool, like ETS320
3. Thermal camera for robotics and UAVs, like FLIR Vue


Supported outputs:
1. 320x240 TFT LCD
2. USB UVC (a common webcam protocol)
3. HDMI 480p/720p (probably even 1080p with Spartan-7 FPGA)
4. AV (NTSC/PAL and many other formats if necessary)
5. Micro SD card (SPI mode + SDIO x4 in future)
6. 3-pin configurable 5V tolerant GPIO to support any low speed interface (UART, SPI, I2C, 1-Wire, PWM, PPM, SBUS...)


Power:
1. Standard Li-ion 18350 battery. Battery life is about 3-3.5 hours (with 1100ma battery), depends on number of active outputs, LCD backlight, etc.
(I'm still working on increasing this value, but I think that reaching 4 hours is a good result.)
2. USB charge (<500ma). USB power is enough for continues camera run with all outputs enabled.
3. External power over aux connector 5.0-35.0V (i.e. up to 8S battery) for UAV mode.


Some features:
1. Adjustable motorized focus system (details later).
2. Supports frame rate up to 60fps (details later).
3. Two standard 1/4" metal mounts on the top and bottom.

More photos/videos and detailed stories of each side of this device will be later too.


Repository:

Follow this project on GitHub: https://github.com/OVGN/OpenIRV

Some sources are not available right now, but I have plans to publish them a bit later.


For beta-testers, developers and backers:

Consider to email me if you would like to be a beta-tester, developer or just a backer.
It would be great if you keep mail header like this: "OpenIRV.Developer/Tester" or "OpenIRV.Backer" or etc...
I promis not to publish your email anywhere else and do not send mails, that are not relevant to OpenIRV project.
I always reply letters. You can easily find my email in attached images

[Cat]

Awesome, this looks very promising
I'm looking forward to learn more about your project, especially how you interfaced the µbolometer and implemented UVC.
Didn't think it would be possible to get a motorized focus inside the tiny housing.
Keep up the good work!

[Fraser]

What a superb project, well executed and very professional in its appearance    

As I own one of these cameras I will await further details with GREAT interest 

Fraser

[zrq]

Quite interested in the interface with the sensor and the reverse engineering. Can't wait to see the code! (and hopefully writeup)

[VGN]

Awesome, this looks very promising
I'm looking forward to learn more about your project, especially how you interfaced the µbolometer and implemented UVC.
Didn't think it would be possible to get a motorized focus inside the tiny housing.
Keep up the good work!

Thank you! Looking ahead, I used FX2LP for USB. FPGA is streaming data over 8-bit bus. As a start point I used this Cypress project: https://community.cypress.com/docs/DOC-14406. More details further... With regard to the focus system, first I didn't believe too)

What a superb project, well executed and very professional in its appearance    
As I own one of these cameras I will await further details with GREAT interest 

Wow, many thanks!   As you are owner of one of these cameras, I hope for your help with testing

Quite interested in the interface with the sensor and the reverse engineering. Can't wait to see the code! (and hopefully writeup)

Hi! The interface with the sensor is, obviously, the most tricky moment. In fact, there are a lot of things that I still don't understand, but I hope for community help.

[conmega]

I also have an NV3 and was working on reverse engineering efforts to interface with the original circuitry.
I just got some boards I made for interfacing to the NV2 going out to some friends to help with development/testing. Although its mostly used as a test platform not such a professional looking project as yours. I look forward to your progress and wish you luck!

[tmbinc]

Very cool project!

Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?

Can't wait to see more details!

[VGN]

I also have an NV3 and was working on reverse engineering efforts to interface with the original circuitry.
I just got some boards I made for interfacing to the NV2 going out to some friends to help with development/testing. Although its mostly used as a test platform not such a professional looking project as yours. I look forward to your progress and wish you luck!

Hi! I think we could colaborate for better result. I don't have NV2, as I know, this is a previous generation camera, it has an internal shutter (behind the lens). Sure you know, there is a great work about NV2: https://debugmo.de/2018/12/autoliv-nv2-teardown/

Very cool project!
Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?
Can't wait to see more details!

Hello! Thanks much! Yes, you are right. I refused using original circuitry in my project, because it may not survive, when the front window is broken. Also original electronics limits some opportunities.

[Max Planck]

Bought some time ago an NV3, but never had time to work on it.

Thus, I'm really impressed by the project.
Are you using just the FPA or also some parts of the NV3?

Max
 

[VGN]

Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?

Thus, I'm really impressed by the project.
Are you using just the FPA or also some parts of the NV3?

Thanks you! Ok, I thinks this will be the start point of the story.

I have three cameras, two of which have cracked front windows (btw cost me under 100$ for each). I didn't expect that condition of the most critical parts, i.e. sensor and lens will be perfect. In fact all three sensors are alive with amount of bad pixes <1%.



As you know, the front window is very fragile. As soon as it catches a stone from the road and cracks, your camera will soon stop working, because of water. Some water damage pictures:


Nevertheless the sensor itself in located in the center of the camera module, and incomming water do not reach it, as the water level do not rise above the bottom edge of the broken window. Perhaps that's why both sensors of drowned cameras are working. Of course you should be carefull, there is no 100% garanthy that sensor will work, but at the same time the probability is quite high. The shutter motor can also be easily replaced if necessary.

You can also see a lot of small dents at the surface of the windows. But the surface of all lens are in perfect condition. I think this is because lens is located very deep inside of the enclosure and also protected by broken window. Anyway that gives a chance to bring alive very poor cameras.


Here you can see parts that I have reused from the original camera:


The front window and cap is optional. I don't really use them, because it is an additional obstacle for IR radiation and the device itself is not water or dust protected. It is designed mostly for home and limited outdoor usage.

[Fraser]

So if it is just the casing, lens, FFC flag and Microbolomter PCB that is used in your design, presumably this project could be adapted to the earlier NV2 model ?

You have basically built a thermal imaging camera from scratch which is a significant achievement when you do not have the datasheet for the microbolometer or its recommended bias voltage values. Building the back end video processing package and firmware is a further very impressive achievement. You have skills  Much respect for you 

Fraser

[VGN]

So if it is just the casing, lens, FFC flag and Microbolomter PCB that is used in your design, presumably this project could be adapted to the earlier NV2 model ?

Yes, you are right. I believe it can be adapted to any camera, that needs deep raw image processing.

You have basically built a thermal imaging camera from scratch which is a significant achievement when you do not have the datasheet for the microbolometer or its recommended bias voltage values. Building the back end video processing package and firmware is a further very impressive achievement. You have skills  Much respect for you 

Thanks much! Bias values were picked by hand, I'm still looking for a way to calculate this values properly. Do you have any knowleges about that? After tons of experiments I found out that this is 6-bit value per pixel and could properly feed them to FPA core.

[Fraser]

The bias values are normally provided by the fabricator of the microbolometer and are unique to each one. The user of the microbolometer then uses these values as a start point from which they are fine tuned for the cameras particular application, temperature range etc. The bias voltages can be fine tuned for best NeTD. Sadly I do not have any bias voltage values for FLIR microbolometers.

Fraser

[Treehouseman]

I've been thinking that it could be possible to extract the bias values from the flash chips, I would imagine it's just a semi standardized mapping that if you compare 2 camera's flash images and see what the differences is it may stand out. It would then be a task to see how they're storing it but with the NV2 given that we have the code from it's softcore, the softcore may load in the values, and that could tell us how they're stored. Especially if they keep the same structure they used for debugging where you could also send it values manually for initial calibration.

While we don't currently have the program from the nv3 it's possible it uses the same data encoding. On top of that we could also see how it's used in a Flir E4 or a Flir E40 (NV3 and NV2 respectively) and see if it's the same thing. I would assume they just drop the same table of values in.

[VGN]

Well, there are a lot of things to tell about, so for simplicity I'm going to move step by step from outside to inside.

The whole enclosure consists of four parts:
1. Top (where the display is mounted)
2. Middle
3. Bottom
4. Original black casing

All parts were developed for CNC machining, but I decided to try SLA 3D printing for prototype. Of course, the quality is not super, there are some wracks at the surface and edges. Anyway I'm sure CNC machining with further casting plastic in silicone molds will give much better quality.



The bottom:



The middle:



The metal mounts is nothing but a 1/4" to 3/8" thread adapter. Fitting the adapter from the inside prevents possiblity of unwanted unscrew:



The top:



Finally I had to increase the lens hole a little to prevent field of view narrowing, because previously I had to raise this black case up to place the focus mechanics.
There is a round boarder that helps to make a neat cutout. This cutout will still allow you to fit the front window if you need it, but maybe you will also have to increase the hole of the window cap too.

[Fraser]

Great to read this thread as it gradually reveals the work you have done. Loving this 

Fraser

[Max Planck]

First technical question.
I know, it is maybe to early at this stage, but have you taken into account the heating of your camera enclosure during operation (electronics, battery, external heat sources) and its temperature changes on camera (FPA) operation?

Max

[VGN]

Great to read this thread as it gradually reveals the work you have done. Loving this 

Thanks a lot!

First technical question.
I know, it is maybe to early at this stage, but have you taken into account the heating of your camera enclosure during operation (electronics, battery, external heat sources) and its temperature changes on camera (FPA) operation?

Yes, it's taken into account. As you know, all FPA pixel are different, they all have different response to the same amount of IR radiation. So to solve this problem and make the image uniform we should determine a gain and an offset parameter for each pixel. Gain parameter changes with the camera temperature quite slowly, whereas the pixel offset changes at a temperature difference of ~0.2°C. To compensate this temperature drift FFC (Flat-field correction) procedure is initiated. We use external shutter to close the front of the lens and assuming that shutter surface is uniform, we perform recalculation of offset parameter, improving the image. You can run FFC manually any time you want to improve the image quality or enable automatic mode, when firmware monitors enclosure temperature by integrated temperature ICs and initiates FFC periodically.

[VGN]

The display module.



This is the biggest LCD that I could find to fit in 60x62mm dimentions. Also manufacturer promises long term supply for this part (10 years since 2018).

Brass thread inserts are glued into plastic by cyanoacrylate. That looks ugly, that's why plastic molding is preferable for manufacturing.

Buttons cover is made of platinum cured silicone with black pigment. It's pretty reliable and looks good. As you see the button's plunger is quite small, so I decided to place additional 0.3mm plastic plunger that will allow to increase button's press area. Plastic plungers were cutted by hand and looks also ugly, but laser cut will fix that.

There is also a small bug, the buttons board should have additional mounting hole in the center. Will be fixed in future revision.

Few words about electronics. The CLC filter reduces power noise from AUX port. DC-DC converts 5.0V-35.0V to 4.5V. Unfortunately, there was no enough space to fit this regulator on the mother board, that's why it is here. All inductors are shielded to reduce possible EMI noise. LCD backlight is powered by constant current regulation IC (not PWM) also to minimize the power noise level.




Feel free to ask questions and criticize the design, that may help to make it better.

[newex]

The amount of work you have done is amazing.
I think it makes no sense to criticize the design, because we do not see the schematic diagram, but it looks great and professional.

[Vipitis]

I am more than impressed by the effort so far. Wish I had the knowledge, experience and tools to mount such a project. But really shows what can be done... and motivates me to move on with my project (in 14 days.)

[Fraser]

I am loving this thread. I very much look forward to seeing future updates. Your camera project looks very professional and not unlike what I have seen in R&D prototyping. Very professional in all respects.

I might add that even with the skills to create the mechanical and electronic design, that is not enough for such a project, as I well know. The fact that you also have the skills required to write the code for the hardware is very impressive

Fraser

[Max Planck]

Yes, it's taken into account. As you know, all FPA pixel are different, they all have different response to the same amount of IR radiation. So to solve this problem and make the image uniform we should determine a gain and an offset parameter for each pixel. Gain parameter changes with the camera temperature quite slowly, whereas the pixel offset changes at a temperature difference of ~0.2°C. To compensate this temperature drift FFC (Flat-field correction) procedure is initiated. We use external shutter to close the front of the lens and assuming that shutter surface is uniform, we perform recalculation of offset parameter, improving the image. You can run FFC manually any time you want to improve the image quality or enable automatic mode, when firmware monitors enclosure temperature by integrated temperature ICs and initiates FFC periodically.

Yes, you have two things, NUC and thermal dirft caused by several factors. I was asking about the latter. You are correcting it using a shutter. My question was about the dynamics of that process, i.e. how often the shutter has to be activated to keep a decent image quality/temperature measurement accuracy.

Knowing about some of the challenges you were and are facing, I'm impressed.

Max 

[VGN]

The amount of work you have done is amazing.
I think it makes no sense to criticize the design, because we do not see the schematic diagram, but it looks great and professional.

Thank you! Well, yes, I realize that this hard to criticize the design without the schematics, but I mean, maybe you have some global ideas that do not require schemes currently. The problem is that I'm going to make quite a lot of changes for the next hardware revision and I think that it would be better if I publish the very last schematics. I believe we will have enough time to discuss and make changes to design if necessary before I send new files for manufacturing.

I am more than impressed by the effort so far. Wish I had the knowledge, experience and tools to mount such a project. But really shows what can be done... and motivates me to move on with my project (in 14 days.)

Thanks much! I wish you good luck with your project!

I am loving this thread.
...

Many thanks, Fraser!

Yes, you have two things, NUC and thermal dirft caused by several factors. I was asking about the latter. You are correcting it using a shutter. My question was about the dynamics of that process, i.e. how often the shutter has to be activated to keep a decent image quality/temperature measurement accuracy.
Knowing about some of the challenges you were and are facing, I'm impressed.

Thank you, Max! Basically one should perform FFC every time when camera's temperature changes for a some threshold value up or down. Also there is no need to perform a FFC if temperature is stable, the image quality will not get worse in time. I can't exactly say how often the shutter has to be activated, as it depends on the environment temperature, how fast the camera heats up and the temperature change threshold value. I don't think that this a point of worry, until you use it on UAV, because FFC causes an image freeze for a short period of time (~1 sec). There are two ways to mitigate this problem. First - you can initiate FFC yourself, second - the firmware can display a countdown before FFC to help the pilot get ready.

[VGN]

Peripheral board (P-board):



Some descriptions:



This board do not process image at all, but provides output path to different interfaces, manages battery power, powerpath logic and some other stuff. D-board (display) and B-board (buttons) are connected to it through 12-pin and 5-pin JST connectors. You can also see a 6-pin JTAG header that is used for FPGA debug on the M-board (main), which I will show futher.



Some device safety features:

• All IOs at HDMI, USB and 7-pin AUX port are ESD protected.
• USB 5V power line has overcurrent (>500ma) and overvoltage (up to 20V) protection.
• AV output has an output buffer with short-to-ground and short-to-battery (up to 18V) protection.
• AUX external power line has high side reverse polarity protection. (high side protection type is probably not the best choice...)
• Three GPIOs at the 7-pin AUX port that goes directly to FPGA are protected by 5V tolerant bidirectional buffers.
• Battery reverse polarity protection. Magic smoke will not appear even if you install battery wrong for a continues period of time.
• Battery is also protected by a special charge/discharge voltage/current protection IC like in factory protected Li-ion batteries. I refused using protected Li-ions, as their length can vary too much from manufacturer to manufacturer, you simply won't be able to insert the battery into holder, meanwhile dimentions of unprotected batteries are much more predictable.

Other features:

• RTC provides timestamps for SD card filesystem.
• Temperature sensor.
• Power reset button.