OpenIRV. ISC0901B0 (Autoliv NV3, FLIR E4/5/6/8) based opensource thermal camera

[VGN]

Here some more images, captured right by camera. Original images were in .bmp format, as this is the easiest way for image capturing, but I had to convert it to .png, as eevblog forum do not support .bmp attaches. Of course in future I plan to add png support right on camera, as bmps are much larger.

P.S. videos are comming soon!

[Fraser]

You cannot beat a decent microbolometer coupled with a good lens 

At least you get a decent, low noise, image to process Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

Fraser

[ajw107]

Thank you! I'm really very glad, that you and other people are interested in this project, that additionally motivates to continue. Yes, I have plans to make a some kind of kit, that anyone can buy. This kit will include everything you need, except the thermal camera, sd card, battery (due to some problems with air transportation of batteries). Later I will determine the exact kit contents. There is no need to buy a FPGA for this project. The design of this camera is developed in a way that you will not need even a solder iron to put all parts together. I will also make efforts to keep the functionality and price competetive with other brands, though that will be not easy.

That is great news, as I'm no good with a soldering iron (rubbish eyesight), although i will keep on trying to give it a go, much to the deprement of my fingers.  I have to admit, I have also just gone and bought a new SLA printer (Kelant S500, bit of a gamble as no reviews - but on paper is sounds good for the price), as my old ANET8 has long since given up the ghost and I've been itching to try an SLA for years.  The photos are great by the way, is that a brain cross-section in photos 11-14? I know it can't be, but it looks like it, he he!  Looking forward to being able to order the curcuitrs when they become available, so excited, like a kid with a new toy.

[ajw107]

Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).

[VGN]

Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).

This thread is actually the guide.
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...

[VGN]

I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...

You cannot beat a decent microbolometer coupled with a good lens 
At least you get a decent, low noise, image to process Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

The lens definitly matters. I don't have Flir E8 right now, but I remembered that earlier I have made a some photos at home. Here is the same bathroom with OpenIRV and FLIR E8 (hacked E4). Of course, shooting time and some other conditions are different:

[VGN]

I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...


Earlier I showed some images, shooted in good conditions, when the scene tempretature difference is high. As you could see, there is no noticeable noise.
Now, let's look at abilities of this camera in poor conditions.

First, so that you can better understand what's going on, here is a photo of bicycle at the wall with a cup of warm water. The room temperature is +25.5 °С , the water in cup is +29.5 °С (measured by external thermometer). Temperature difference of the coldest and hottest objects is about 4-5°С. As you can see, the image is quite good, further we will analyze this scene without a cup of warm water.




The same scene without a cup of warm water. This gif is a sequence of images shooted with a small time gap in a raw mode (no any kind of filtering):



As you can see, the noise gets out, as temperature range is probably about ~1°С. The noise pattern is complex and quite specific. After reading tons of literature and some analysis and I came to think that there are two main noise components:
1. Row noise. This is a temporal noise, i.e. random fluctuations of the whole pixel line, that randomly changes every moment.
2. Column noise. This is a FPN (fixed pattern noise) that changes quite slowly and cannot be fully removed even by FFC (flat field correction) procedure.


Earlier I have developed a special averaging module that helps to average a sequence of multiple frames comming out of the sensor. Max possible number of averaged frames is 262144, though reasonable values are 2, 4, 8, 16, 32, 64, 128. Signal averaging is a common technique, intended to increase the strength of a signal relative to noise. So, let's try to use this module with different averaging values:



We can see that starting from x8 value the row noise disapears, that confirms the nature of row noise. At the same time column noise is still present, averaging do not help to neutralize this type of noise, we need something more.


Few days ago I found a way to remove this column FPN in frequency domain by FFT (fast fourier transform) and a custom filter. As a test scene I decided to use some uniform object. Here you can see how it works. Image from camera (uniform body) is on the left, FFT of this image is on the right. Black lines on the FFT plot are the actual filter function, that "cuts out" the noise components. Here is the link to the online FFT tool that I used for experiments: https://www.ejectamenta.com/Fourifier-fullscreen/




At last I decided to test both averaging and FFT filters at the complex scene with a lot of details:



Also keep in mind that this experiments are done on equalized 8-bit image. I think that final image quality will be a little better if I apply FFT filter to the raw 14-bit pixel frames.


I don't want to start developing this FFT filter right now, it will be easier to develop it for new hardware and new FPGA. Though averaging is a good filter, it reduces the frame rate, that's why I should implement something else for row noise, of maybe FFT will be enogh, the time will show. At least I know how to deal with this noise. If you have good ideas and suggestions for image filtering, please don't keep in secret


P.S. This is how lepton 80x60 "sees" this bicycle: 

[zrq]

For the similar noise I noticed on Xtherm cameras, I wrote a routine to reduce the single column noise by filtering in the temporal space.

Code: [Select]

merit = np.abs(cv2.Laplacian(cv2.GaussianBlur(imgFlt, (3, 3), 3), cv2.CV_32F)) / 65536
merit = 1 / pow(merit + 0.005, 3)
leftErr = imgFlt[:, 0:leftHalf] - cv2.GaussianBlur(imgFlt[:, 0:leftHalf], (1, 5), 10)
horiLeftAcc = np.average(leftErr, weights=merit[:, 0:leftHalf], axis=1)
rightErr = imgFlt[:, leftHalf:] - cv2.GaussianBlur(imgFlt[:, leftHalf:], (1, 5), 10)
horiRightAcc = np.average(rightErr, weights=merit[:, leftHalf:], axis=1)
vertErr = imgFlt - cv2.GaussianBlur(img, (5, 1), 10)
vertAcc = np.average(vertErr, weights=merit, axis=0)
for n in range(0, np.size(img, 0)):
    imgFlt[n, 0:leftHalf] = cv2.subtract(imgFlt[n, 0:leftHalf], (k[0] * horiLeftAcc[n])).squeeze(1)
    imgFlt[n, leftHalf:] = cv2.subtract(imgFlt[n, leftHalf:], (k[1] * horiRightAcc[n])).squeeze(1)
for n in range(0, np.size(img, 1)):
    imgFlt[:, n] = cv2.subtract(imgFlt[:, n], (k[2] * vertAcc[n])).squeeze(1)
However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked.

[VGN]

However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked.

I made the same mistake at first)
I also think that parametrizing this filter pattern like this will help to find optimal configuration:




By the way, I found a cool Protoshop plug in for FFT noise reduction.
http://ft.rognemedia.no/

I tested it on bicycle image, drawing that cross filter pattern by hand. Now no image crop, no averaging, only pure FFT filter.

Before:



After:



You can see some luminance fluctuation in filtered image, I'm sure that this is caused by changes in histogram equalization, caused by noise peaks. That's why we should apply this filter on 14-bit raw image before histogram gain correction. Anyway, there is almost no high frequency row/column noise and that is really good!

[ajw107]

This thread is actually the guide.
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...

You know, I thought that as soon as I pressed the post button (that's what I get for asking questions when I'm half asleep).  In my head there was some magical M12 CAN -> USB connector, he he!  Any news on when things will be ready for us to try (no rush, just over excitment, like a child with a new toy)!

[Hydron]

These noise issues are present on other microbolometers too - e.g. the ULIS 17u 640x480 FPA has a well known issue with column noise, and there are some filtering techniques that have been tried to improve it (some suggested by ULIS themselves I think). I'm not sure whether I have any info about these, but if I do and they are able to be distributed (unfortunately unlikely) I'll do so.

[VGN]

Hello!

Sorry for the delay, I was working at the next hardware revision. I have a lot of progress and updated. But first, the video capture, that I have promised long ago:



I doubt that youtube will ever show this video in original quality. So, for those of you, who would like to see the original quality and frame rate, here is the link to my google drive (video file is about 180MB): https://drive.google.com/file/d/1QfK6TBKxTnJjb9-Vjo1R4MsX56F_mhFq/view?usp=sharing
Google drive will suggest you to watch it online in browser, but you should better download it.

Video was captured over USB. Note that this thermal camera do not record the sound. The sound was added later.

[Jenny]

Hi, I’m so amazed (like others also do) at how incredible work you have done. 
Just a quick question regarding to the last video: I realized you have motorized focus on the camera? Is it auto or manual? I found it is a bit slow, but stop at the focal point without overshoot. Is it phase detection or even laser assisted?
Again, it’s really unbelievable how you done such work, in a short time...

[VGN]

Hi, I’m so amazed (like others also do) at how incredible work you have done. 
Again, it’s really unbelievable how you done such work, in a short time...

Thank you! I'd not say, I have done it in a short time..

Just a quick question regarding to the last video: I realized you have motorized focus on the camera? Is it auto or manual? I found it is a bit slow, but stop at the focal point without overshoot. Is it phase detection or even laser assisted?

Yes, the focus is motorized, you can find focus system PCB photos in my previous posts. There are two modes, manual and auto. On the video, you saw a manual mode, I changed the focus myself. Autofocus fuction is under development now. It is based on image sharpness detection. Right now it is implemented in software and works too slow and not precize. I definitly should make a hardware acceleration of image sharpness calculation. In this case I will be able to detect sharpness gradient change at the highest frame rate, and accidental sharpness change due to the hand shake will not confuse the algorithm, I hope.

Regarding to the speed, yes it is slow now, because I have limited the RPM. The focus motor is rated for 3.0V, but it is running at ~2.0V now. Both shutter and focus motors are powered almost directly from Li-ion battery, which voltage is variable (4.2V-3.0V) and a little higher than motor recommended voltage. As I don't want to overvoltage the motor power, I'm using PWM to limit the voltage. Further I'm going to adjust the PWM duty cycle according to the battery voltage to keep it at 3.0V.

There is also another feature. We have 4 lens turns to adjust the focus. The focus distance changes non-linearly with the lens position, and that is very good, because a single turn from the fully closed position allows to cover focus distances from ~30cm to infinity. Lens movement under single turn will be pretty quick, thought for macro mode you will have to make few other turns.

Forgot to say, there is one more way to increase the focusing speed by using motor with smaller gear ration. Right now reductor ration is 1:136. But there same type of motors with 1:26 gearbox, with higher RPM. If torque will be enough, it will be worth replacing the motor.

[ajw107]

I have to admit, I was getting a little worried when we hadn't heard from you in a while, so pleased it is going well.  The video quality is incredible, it seems to have come on leaps and bounds from the still very good bike photos we had earlier.  Wonder if I could ask a few questions:
1)  Will the FPGA be able to handle the thermal data processing and the auto focus and keep the frame rate?  I'm not familiar with the Spartan-6 board?
2) Is there an ETA on when we can get a look at some of the code and stl files?  I'm just being impatient with excitement, I know.  But The code will be fascinating to look through for the image processing alone, and the stl files will be a good test for my printer (whenever it eventually arrives, grrr).
3) I'm not entirely sure where abouts you are in your project roadmap, it seems you are getting close to a finished product, just requiring some some tweaks (sometimes the phase that takes the longest time, but hopefully everyone on here can help once it goes on github).
4) Any idea of how much the kit minus the nv3 will cost?  I'm moving house at the moment, so counting the pennies, and want to make sure I put some money aside.  Even better if you could say some thing like the circuitry will cost about X and the housing/gears/internal parts should be about Y, etc.

Anyway, as I say, magnificent work, it really is impressive what you have accomplished.

[VGN]

I have to admit, I was getting a little worried when we hadn't heard from you in a while, so pleased it is going well.

Sorry for this, I will try not to make long delays or at least warn about that.

1)  Will the FPGA be able to handle the thermal data processing and the auto focus and keep the frame rate?  I'm not familiar with the Spartan-6 board?

New hardware will be based on newest Spartan-7. Of course, FPGA will be able to handle the thermal data processing, autofocus and many other things simultaneously. FPGA is very good with paraller processing.

2) Is there an ETA on when we can get a look at some of the code and stl files?  I'm just being impatient with excitement, I know.  But The code will be fascinating to look through for the image processing alone, and the stl files will be a good test for my printer (whenever it eventually arrives, grrr).
3) I'm not entirely sure where abouts you are in your project roadmap, it seems you are getting close to a finished product, just requiring some some tweaks (sometimes the phase that takes the longest time, but hopefully everyone on here can help once it goes on github).

Don't get me wrong, this is my first open source project. Though I have developed quite a lot of different devices in a small teams with private repositories, I don't have an experience with open source and wide community. I'm not sure I know the proper way of organizing such kind of projects. Is it better to make a public repo from the very beginning or keep it private for those of you, who would like to collaborate and reveal sources with the first batch of devices? I've always thought about it. Maybe you or anybody else can give an advice?

Also I didn't expect that I will make so many changed for the new revision, this is another reason that I haven't published schematics yet... . Tomorrow I will tell about most of new features.

Regarding to the technical side of the project, in general terms, this is my plan for nearest future:
1. Finish new hardware revision schematics (will be finished in a week).
2. Purchase components and new PCBs (shipping is about 2-3 weeks).
3. Rewrite the HyperRAM memory controller for Spartan-7 and publish it as a separate project.
4. By this time I hope to get new PCBs and component and can assembly everything.
5. Start porting the design to new hardware.

If everything is ok at this step and I'm sure that hardware is working properly, I could make a first small batch of devices for developers and testers. After that I think we will be ready for larger production. This is a matter of discussion.

4) Any idea of how much the kit minus the nv3 will cost?  I'm moving house at the moment, so counting the pennies, and want to make sure I put some money aside.  Even better if you could say some thing like the circuitry will cost about X and the housing/gears/internal parts should be about Y, etc.

I will provide this information as soon as I get the final BOM. But I'm going to keep the price competitive to HT-301, TE-Q1, HT A2, FLIR C2 and other "cheap" thermals, though OpenIRV is going to be more powerful and multifuctional. Also keep in mind that the batch size makes difference on the final price.

[railrun]

If you are looking for tester I would be happy to be a tester.

[bicycleguy]

@VGN
Looks very interesting.

Do you know if a 4H0980552 is an acceptable version.  Note the H instead of G.  How would I find out what that means?

thanks

[VGN]

If you are looking for tester I would be happy to be a tester.

I'm very glad, thank you! That will be cool if you and all other people, who would like to be a tester or even a developer of the first kit batch, email me (view my profile for email).

Do you know if a 4H0980552 is an acceptable version.  Note the H instead of G.  How would I find out what that means?

I don't know what actually each letter means, but according to the photos, 4H0980552 is the previos generation (NV2). This project initially was designed for NV3, but I have plans to make hardware in a way to support more types of sensors.

[tmbinc]

The easiest take-away is the connector - it's always this HSD ("weird automotive never seen before") connector, and either there are two additional pins next to it (NV2) or not (NV3). NV2 has Power (separate), CAN and LVDS so it needs 6 pins, whereas NV3 has Power and this "weird automotive never seen before bidirectional highspeed interface that's neither CAN nor LVDS but can replace both", and hence only needs 4 pins.

The case is also distinctively different, but harder to describe/see.

[VGN]

The easiest take-away is the connector - it's always this HSD ("weird automotive never seen before") connector, and either there are two additional pins next to it (NV2) or not (NV3). NV2 has Power (separate), CAN and LVDS so it needs 6 pins, whereas NV3 has Power and this "weird automotive never seen before bidirectional highspeed interface that's neither CAN nor LVDS but can replace both", and hence only needs 4 pins.
The case is also distinctively different, but harder to describe/see.

Exactly!

[bicycleguy]

Thanks for the help.  Had a seller on eBay tell me they were the same.  He said the 4H was from an Audi A8 instead of A7.  Luckily I saw your quick responses and hadn't paid yet. 

thanks.

[tmbinc]

Well for NV2 we have https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/, but VGN's work of course is far more impressive.

[VGN]

Well for NV2 we have https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/, but VGN's work of course is far more impressive.

tmbinc, btw I have to notice that I saw your investigation of NV2 on debugmo.de long ago. That is an incredible work that inspired me to work on NV3! I'm a newbie on this forum, I haven't remembered your nick-name, so I only recently realized that was you, haha)) 
Long story short, very glad to meet you!

[VGN]

I'm finishing new hardware revision and want to tell you about main changes:

1. Most of the changes are refered to M-board (main). I have replaced old Spartan-6 XC6SLX45 with new Spartan-7 XC7S50.

2. I decided to add more RAM. I found new HyperRAM part which is twice cheaper and +20% faster than the previous one. The reason of adding more RAM is that probably image filtering with FFT will require more memory bandwidth and I'm afraid that single channel RAM will not be enough. Anyway it is always better to have a footprint for an additional memory rank, than do not have it at all.

3. From some point I realized that it is worth to support or, better say, give a chance to support more types of sensors than only NV3's one.

NV3's sensor ISC0901B0 is connected to the main board over 40-pin connector. There are:

• 1 pin is connected to sensor frame (chassis)
• 2 pins for bias power (11.0v)
• 2 pins for core power (3.3v)
• 2 pins for io power (2.5v)
• 6-pins to control the sensor. clk, ena, cmd, bias, data_odd, data_even. Details further.
• 2 pins are NC (not connected to sensor die and anywhere else)
• 25 pins (remaining) are ground

Well, leaving 25 pins for ground is ridiculous, that's why I have developed a new backward compatible pinout that will use some ground pins as IOs and connected them to FPGA. If this extended IOs will be configured as input at FPGA side, the direct path to ground over ISC0901B0 board will not stress FPGA's bank.

Generally now we have 18 single ended IOs that are connected to a single FPGA bank. The bank power can be configured in safe range (0.6v - 3.3v) over DC-DC's feedback resistor divider. Other board curcuits have separate power, that's why IO voltage change will not affect anything else. I hope these amount of IOs will be enough to connect any sensor. Probably it is worth to route some of this IOs in differential line manner, but I'm not sure. Any ideas or suggestions are welcome.

Also two NC pins I have connected to system power (battery rail) as an additional power path.
Bias and core power rails are also configurable or can be disabled if desired.

4. I have reworked whole power system, removed 2 LDOs and improved efficiency of some power rails.

5. You can also see some strange 10-pin blue connector. There are 7 IOs (3.3V fixed), ground and two power rails (regulated 3.3V + battery rail). It can be used for any general purpose, but I have one quite interesting task for this connector, details will be furter.


Regarding to the support of different sensors. We have a base part and any other sensor can be connected over special adapter board. Optionally we will also need a cover to protect the sensor. The most common sensors that I think is possible to support, except NV3's are NV2 sensor (ISC0601B), probably Seek Thermal Mosaic Cores and of course Lepton2/3 (though this hardware for such a tiny core is an overhead...)