Author Topic: OpenIRV. ISC0901B0 (Autoliv NV3, FLIR E4/5/6/8) based opensource thermal camera  (Read 3752 times)

0 Members and 1 Guest are viewing this topic.

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Here some more images, captured right by camera. Original images were in .bmp format, as this is the easiest way for image capturing, but I had to convert it to .png, as eevblog forum do not support .bmp attaches. Of course in future I plan to add png support right on camera, as bmps are much larger.

P.S. videos are comming soon!
« Last Edit: July 27, 2020, 07:26:11 pm by VGN »
 

Online Fraser

  • Super Contributor
  • ***
  • Posts: 9543
  • Country: gb
You cannot beat a decent microbolometer coupled with a good lens  :-+

At least you get a decent, low noise, image to process :) Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

Fraser
 

Offline ajw107

  • Newbie
  • Posts: 4
  • Country: gb
Thank you! I'm really very glad, that you and other people are interested in this project, that additionally motivates to continue. Yes, I have plans to make a some kind of kit, that anyone can buy. This kit will include everything you need, except the thermal camera, sd card, battery (due to some problems with air transportation of batteries). Later I will determine the exact kit contents. There is no need to buy a FPGA for this project. The design of this camera is developed in a way that you will not need even a solder iron to put all parts together. I will also make efforts to keep the functionality and price competetive with other brands, though that will be not easy.

That is great news, as I'm no good with a soldering iron (rubbish eyesight), although i will keep on trying to give it a go, much to the deprement of my fingers.  I have to admit, I have also just gone and bought a new SLA printer (Kelant S500, bit of a gamble as no reviews - but on paper is sounds good for the price), as my old ANET8 has long since given up the ghost and I've been itching to try an SLA for years.  The photos are great by the way, is that a brain cross-section in photos 11-14? I know it can't be, but it looks like it, he he!  Looking forward to being able to order the curcuitrs when they become available, so excited, like a kid with a new toy.
 

Offline ajw107

  • Newbie
  • Posts: 4
  • Country: gb
Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).
This thread is actually the guide. :D
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...



You cannot beat a decent microbolometer coupled with a good lens  :-+
At least you get a decent, low noise, image to process :) Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

The lens definitly matters. I don't have Flir E8 right now, but I remembered that earlier I have made a some photos at home. Here is the same bathroom with OpenIRV and FLIR E8 (hacked E4). Of course, shooting time and some other conditions are different:
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...



Earlier I showed some images, shooted in good conditions, when the scene tempretature difference is high. As you could see, there is no noticeable noise.
Now, let's look at abilities of this camera in poor conditions.

First, so that you can better understand what's going on, here is a photo of bicycle at the wall with a cup of warm water. The room temperature is +25.5 °С , the water in cup is +29.5 °С (measured by external thermometer). Temperature difference of the coldest and hottest objects is about 4-5°С. As you can see, the image is quite good, further we will analyze this scene without a cup of warm water.

[attachimg=6]



The same scene without a cup of warm water. This gif is a sequence of images shooted with a small time gap in a raw mode (no any kind of filtering):

[attachimg=1]

As you can see, the noise gets out, as temperature range is probably about ~1°С. The noise pattern is complex and quite specific. After reading tons of literature and some analysis and I came to think that there are two main noise components:
1. Row noise. This is a temporal noise, i.e. random fluctuations of the whole pixel line, that randomly changes every moment.
2. Column noise. This is a FPN (fixed pattern noise) that changes quite slowly and cannot be fully removed even by FFC (flat field correction) procedure.



Earlier I have developed a special averaging module that helps to average a sequence of multiple frames comming out of the sensor. Max possible number of averaged frames is 262144, though reasonable values are 2, 4, 8, 16, 32, 64, 128. Signal averaging is a common technique, intended to increase the strength of a signal relative to noise. So, let's try to use this module with different averaging values:

[attachimg=2]

We can see that starting from x8 value the row noise disapears, that confirms the nature of row noise. At the same time column noise is still present, averaging do not help to neutralize this type of noise, we need something more.



Few days ago I found a way to remove this column FPN in frequency domain by FFT (fast fourier transform) and a custom filter. As a test scene I decided to use some uniform object. Here you can see how it works. Image from camera (uniform body) is on the left, FFT of this image is on the right. Black lines on the FFT plot are the actual filter function, that "cuts out" the noise components. Here is the link to the online FFT tool that I used for experiments: https://www.ejectamenta.com/Fourifier-fullscreen/

[attachimg=3]



At last I decided to test both averaging and FFT filters at the complex scene with a lot of details:

[attachimg=4]

Also keep in mind that this experiments are done on equalized 8-bit image. I think that final image quality will be a little better if I apply FFT filter to the raw 14-bit pixel frames.



I don't want to start developing this FFT filter right now, it will be easier to develop it for new hardware and new FPGA. Though averaging is a good filter, it reduces the frame rate, that's why I should implement something else for row noise, of maybe FFT will be enogh, the time will show. At least I know how to deal with this noise. If you have good ideas and suggestions for image filtering, please don't keep in secret ;)



P.S. This is how lepton 80x60 "sees" this bicycle:  ;D

[attachimg=5]
« Last Edit: July 30, 2020, 11:27:18 am by VGN »
 

Offline zrq

  • Contributor
  • Posts: 25
  • Country: cn
For the similar noise I noticed on Xtherm cameras, I wrote a routine to reduce the single column noise by filtering in the temporal space.

Code: [Select]
merit = np.abs(cv2.Laplacian(cv2.GaussianBlur(imgFlt, (3, 3), 3), cv2.CV_32F)) / 65536
merit = 1 / pow(merit + 0.005, 3)
leftErr = imgFlt[:, 0:leftHalf] - cv2.GaussianBlur(imgFlt[:, 0:leftHalf], (1, 5), 10)
horiLeftAcc = np.average(leftErr, weights=merit[:, 0:leftHalf], axis=1)
rightErr = imgFlt[:, leftHalf:] - cv2.GaussianBlur(imgFlt[:, leftHalf:], (1, 5), 10)
horiRightAcc = np.average(rightErr, weights=merit[:, leftHalf:], axis=1)
vertErr = imgFlt - cv2.GaussianBlur(img, (5, 1), 10)
vertAcc = np.average(vertErr, weights=merit, axis=0)
for n in range(0, np.size(img, 0)):
    imgFlt[n, 0:leftHalf] = cv2.subtract(imgFlt[n, 0:leftHalf], (k[0] * horiLeftAcc[n])).squeeze(1)
    imgFlt[n, leftHalf:] = cv2.subtract(imgFlt[n, leftHalf:], (k[1] * horiRightAcc[n])).squeeze(1)
for n in range(0, np.size(img, 1)):
    imgFlt[:, n] = cv2.subtract(imgFlt[:, n], (k[2] * vertAcc[n])).squeeze(1)

However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked. :-\
 
The following users thanked this post: nikitasius

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked. :-\
I made the same mistake at first)
I also think that parametrizing this filter pattern like this will help to find optimal configuration:

[attachimg=3]



By the way, I found a cool Protoshop plug in for FFT noise reduction.
http://ft.rognemedia.no/

I tested it on bicycle image, drawing that cross filter pattern by hand. Now no image crop, no averaging, only pure FFT filter.

Before:

[attachimg=2]

After:

[attachimg=1]

You can see some luminance fluctuation in filtered image, I'm sure that this is caused by changes in histogram equalization, caused by noise peaks. That's why we should apply this filter on 14-bit raw image before histogram gain correction. Anyway, there is almost no high frequency row/column noise and that is really good!
« Last Edit: July 30, 2020, 11:03:38 am by VGN »
 

Offline ajw107

  • Newbie
  • Posts: 4
  • Country: gb
This thread is actually the guide. :D
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...
You know, I thought that as soon as I pressed the post button (that's what I get for asking questions when I'm half asleep).  In my head there was some magical M12 CAN -> USB connector, he he!  Any news on when things will be ready for us to try (no rush, just over excitment, like a child with a new toy)!
 

Online Hydron

  • Frequent Contributor
  • **
  • Posts: 473
  • Country: gb
These noise issues are present on other microbolometers too - e.g. the ULIS 17u 640x480 FPA has a well known issue with column noise, and there are some filtering techniques that have been tried to improve it (some suggested by ULIS themselves I think). I'm not sure whether I have any info about these, but if I do and they are able to be distributed (unfortunately unlikely) I'll do so.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf