Thermal Camera+ App by Georg Friedrich Superresolution Comparison

[Vipitis]

I wanted to post a comparison to the different options, to get some feedback and also serve as advice for people thinking about a purchase.

First the setup. Camera is the Lepton sensor in my CAT S60 phone on a tripod, in portrait(so what you see is 90° to the side) target is some pipes in our basement. Emissivity is set to 95% and Reflected Apparent Temperature 300K. Palette use is Iron. Images were taken once the superresolution indicator was green.

Issues with this comparison: The scale of the palette and therefore the histogram is not locked! locking the palette and changing anything in settings messes up the shutter and the app needs to be quit and restarted. You can see that the min/max temperature are not consistent.

There are a few settings to chose from. Manual for reference: http://thermcam.binaryblogger.de/manualA/#superres

1. Superresolution: on / off
2. Superresolution sharpening mode: Medium / Harder
3. Upscaling: Bilinear / Nearest Neighbor

***

Attachments:
1. No Superrsolution, Nearest Neighbor
2. Superrsolution with Medium sharpening, Nearest Neighbor
3. Superrsolution with Harder sharpening, Nearest Neighbor
4. No Superrsolution, Bilinear
5. Superrsolution with Medium sharpening, Bilinear
6. Superrsolution with Harder sharpening, Bilinear

[Ultrapurple]

True superresolution relies on small movements (hand shake) - it simply does not work if the imaging device is fixed and stationary on a tripod and looking at a stationary subject.

There's an explanation here that describes how superresolution works to use sub-pixel data from several consecutive but slightly different frames - the differences it uses are either camera shake or other movements such as panning.

Noise reduction, on the other hand, may need a stationary camera (as described in the manual link you give).

[Vipitis]

Given the tripod used, the minimal force applied to touch the capture button, will provide a slight shake that should be sufficient.

Stacking multiple captures graphically with same temperature scale could be done to reduce noise. But they inconsistency of the Lepton over time and FFC will be an issue.

[tomas123]

@Vipitis
Nice work

I wrote in this forum something about the SuperResolution of thermal images.
SR ist not a magic tool with large improvements. Nothing is better than real pixels 

Superresolution is sharpening of the image without false temperature values on contrast edges.
This a bad side effect of sharpening with Gaussian blur.
As sample look at the "cooler" temperature (light yellow rings) around the pins of the last oversharpen images (Gaussian blur) in this post:


Your CAT S60 lepton sensor with (low) 80x60 pixel has no big potential for more details with SuperResolution.
With a lepton 160x120 pixels you have noticeable more potential for more sharpness on fine (!) thermal structures (see video below in full screen, it's the same algorithm/code as in Thermal Camera+ App).

The greatest SR effect by thermal images is the denoise of (live view = video) images with a small (!!) temperature spans.
In your images with a large span of 21 Kelvin by a 2^8 LUT grayscale (=80mK steps) you have by design only little noise.

If you interested in this topic, make 10 normal thermal images and stack the images with photoshop. There you can play about with the technical possibilities.

a sample with lepton 160x120

here is my first result by testing a "real time" super resolution algorithm with OpenCV

left side: Flir One 160x120 Lepton 3
right side: same image with super resolution

https://youtu.be/6-iRTvr0uAw

the youtube compression is really hard, it looks better in reality 

compare this two screenshots from the video above (and don't forget that we have only 160x120 sensor!):


you see the cover of a ZBOX CI323 nano

a sample of 90 (!!) frames of Flir Exx:

macro without macro lens

Testo has a feature called SuperResolution, to stack multiple images for better resolution

I tried the image stacking with some Flir Images.
...
compare the images

a single frame

Code: [Select]

$ exiftool -b -RawThermalImage seq101.fff | convert - -auto-level resize 640x -clut single.jpg

the LQFP64 chip size in the middle is 8x8mm

a single frame sharpen

Code: [Select]

$ exiftool -b -RawThermalImage seq101.fff | convert - -auto-level -resize 640x ../iron-pal.png -sharpen 0,3 -clut single-sharp.jpg

and the stacked image of 90 frames

Code: [Select]

$ convert registax.tif -resize 640x ../iron-pal.png -clut stack.jpg

and now a (over) sharpen stacked image at 300%

look at the vertical chip labeling SMSC®
LQFP64 pin-distance: 0.5mm (16 pins at 8 mm)

...

Testo published a nice paper for SuperResolution (super-sampling and deconvolution)

...

[Vipitis]

I emailed Bullit and asked, if a upgrade is possible. They denied this because the Lepton3 is larger then what they used and they would have to redesign the whole PCB etc.

I only got 80x60 pixels and 2 died already. I want to find out what I can do with it. I am currently exploring many different options in creating more then the standard flir processed images.

the Thermal Camera+ app gives me real pixels and a functional way to lock the palette. I am currently attempting photogrammetry, but need to find a way to mask the min/max and scale from the image so they get ignored by VisualSFM.

The app sadly on captures single images, no sequences, not timers.

[tomas123]

look at my footer
There you find many old hints for post processing of the Flir radiometric jpgs (saved by Flir app).
You can do  this with simple scripts (look palette, photogrammetry eg)

[frenky]

Just to make things clear... Super-resolution and image stacking are two different things.

While image stacking can greatly reduce noise it does not increase resolution.

for super resolution you need to take multiple images that are slightly offset. Then upscale each pixel and align all images.
This way you get extra information about values that are between neighbour pixels of each frame.

[Ultrapurple]

Just to make things clear... Super-resolution and image stacking are two different things. (...)

I completely agree with all of what you say here. You said it far better than I managed!

As we've seen on the Zotac video by Thomas123, true superresolution is capable of really impressive results.

Combining superresolution, stacking and sharpening can give further remarkable improvements. I haven't properly been able to do all three on moving images yet but I'm sure someone clever out there will be able to do so.

[frenky]

I did some super-resolution experiments while trying to improve original SeekThermal.
I mounted Seek module on stepper motor and took photos while motor was rotating:



Result were quite better than original Seek images:

Original:


Many images aligned and stacked to achieve super-resolution:


If you look at IC pins. I would never be able to get such an improvement only with stacking.

I did my aligning by hand & eye in Gimp.

But there are out the box solutions made for astrophotography and could be used for thermal imaging:

AutoStakkert

AutoStakkert!  is all about alignment and stacking of image sequences, minimizing the influence of atmospheric distortions (seeing). Its goal is to create high quality images of the Planets, the Sun, and the Moon, without too much hassle.

https://www.autostakkert.com/wp/download/


RegiStax

Registax 6 software for alignment/stacking/processing of images

http://www.astronomie.be/registax/download.html

[Vipitis]

The last example even has stiching with it.

Panoramic stiching with rotating motion to further, get a larger image.

In the end a solution that does all 3 things would be the goal, and it works in the capturing app.

[tomas123]

But there are out the box solutions made for astrophotography and could be used for thermal imaging:

RegiStax

Registax 6 software for alignment/stacking/processing of images

http://www.astronomie.be/registax/download.html

... as you can see in my postimg above, it works great ;-)

and the stacked image of 90 frames
   $ convert registax.tif -resize 640x ../iron-pal.png -clut stack.jpg

[frenky]

But there are out the box solutions made for astrophotography and could be used for thermal imaging:

RegiStax

Registax 6 software for alignment/stacking/processing of images

http://www.astronomie.be/registax/download.html

... as you can see in my postimg above, it works great ;-)

and the stacked image of 90 frames
   $ convert registax.tif -resize 640x ../iron-pal.png -clut stack.jpg

I didn't notice that but it's good to hear that we can use it for thermal.

Sent from my LG-D855 using Tapatalk

[in.Vitro]

Sorry for OT, but is there a chance for a Video Mode? 

[Ultrapurple]

ThermViewer driving a Therm-App camera has superresolution on video (and still images) - it works remarkably well.

When comparing my regular Therm-App (384 x 288) with ThermViewer superresolution against a Therm-App Pro (640 x 480) without superresolution, the lower resolution imager gave at least as good an image, subjectively, as its higher resolution sibling.

Superresolution like this does require a small amount of hand shake: it simply does not work on locked-off tripod images (you get double-size images, but no rxtra actual resolution).

For an example of offline superresolution video processing, see this video of the Spain-Gibraltar border, which was filmed at 640 x 480 and superresolution processed to 1280 x 960 using Video Enhancer 2.

[Vipitis]

As far as I know - video superresolution is doing continues multiframe superresolution.

so it looks at the past 4 frames. and when your imager does 9hz - you can easily get ghosting artifacts on moving targets like cars or people.

Thermal Camera+ does not support video capture. It also lacks any timer or intervall settings which could subsititude video. My main concern with the app right now is that it burns in the min and max temperature as well as the palette gradient and to archive that - images will get upscaled to 508x640 - either using the Nearest Neighbors setting - basically making every pixel become 4 pixels - or the bilinear upscale by mixing the additional pixels form either a 120x160 supperresolution picture or the 60x80 native picture.

Supperresolution has two sharpening modes - while "Medium" is giving more accurate readings, as well as looking better for NN and bilinear upscale the "Harder" sharpening mode will make fine details and edges more visible by inducing more contrast to some edges and more noise to the overall image (I would love to compare it to Flir DDE which they developped for survaliance applications - using same sensor and same target).

The data burn in basically voids data of a few pixels and therefore should have a setting to be turned off or be moved off the image and into the 28 wide column of pixels for the gradient.

I did not succeed to make a imagemagick script to convert a image out of the Thermal Camera+ app into a .jpg without burned in gradient or data to use for photogrammetry or stitching.
I know that it is possible to somehow extract a raw thermal image from FLIR radiometric .jpeg with some sript found on the forum but I failed to do so as well. and the FLIR firmware upsclaes the images from the CAT S60 lepton - or Flir ONE - to 240x320 "IR resolution" in flir tools which is 2⁴x (= 16x) the effective resolution. As well as to a .jpg resolution of  480x640(which could be due to the to bad resolution and still downscaled and cropped visula image being overlayed in MSX). Flir also applies histogramm equalization and some kind of streching to apply the whole color range of the palette on only the range of temperature shown in the image. This has the result of images capture with the myFLIR app, of the same target but with different background highs and lows will look different even if I set the range of temperature to the same min and max the whole image will look different because factors like gain and gamma can't be changed(It might be that some of those factors are calculated through the parameters in Flir tools: Emissivity, Refl. temp., Distance, Atmosphereic temp., Ext. optics temp., Ext. optics trans. and Relative humidity).

I am still looking for a solution to capture multiple images - by hand, as interval or as 9hz video - using the very same fake color - or a radiometric fashion where I can adjust the fake color palette in post. A solution would be to turn your all images into a raw array of 80x60 pixels and generate a 14bit greyscale that covers the whole sensor output range in 50mK steps. Flir Tools can generate a 320x240 .cvs with temperature values... but this means they read 16 temperatures out of a single Lepton pixel to 3 decimal places in °C and I know how much of that is bullshit in the end.

Right now Thermal Camera+ with Superresolution on "Harder" (or maybe I should do "Medium" instead) and NN zooming with locked palette(which locks the range of the temperatures as well as gamma) and shooting from multiple angles for photogrammetry or getting closer and shooting an array of a few rows with good overlap for stiching in ICE - this is only effective at a large FoV when the target is flat as perspective becomes an issue with planar motion.

Videos and Timelapses made in myFLIR app are bad because locking the temp range of the is broken due to the Shutter going crazy and unless your burn in a spot you can't get any data. not even a min/max and gradient. Flir radiometric video is only part of their high end research cameras.

[tomas123]

Don't forget, that "continues multiframe superresolution" of Thermal Camera+ also reduce the noise in thermal images with low temperature span.

I can confirm the "histogramm equalization" in Flir images. With the 16bit RAW images (inside your thermal jpgs) you can bypass the equalization.
It's simple, see my tutorials.

IMHO a 80x60 Lepton (CAT S60 , Flir One G1 for IOS) is not a good basis for image enhancements.
see my sample images: https://www.eevblog.com/forum/thermal-imaging/flir-e4-thermal-imaging-camera-teardown/msg802245/#msg802245

Flir One G1 Lepton 80x60



Flir One G2 Lepton 160x120 with hacked config file

[Vipitis]

I think if you shoot a still target with a few superresolution(handheld) shots and then stack the images you get rid of more noise.

I have to try and understand all your work with exiftools and learn to use it - but that is a project for next weekend.

[Ultrapurple]

I think if you shoot a still target with a few superresolution(handheld) shots and then stack the images you get rid of more noise.

Yes, you're right. This image was basically done that way - a 10 second video (rather than individual photos) totalling about 250 frames, processed in Registax 6 to produce a superresolution, noise-reduced version. Click the image for more info.



The vertical line in the centre of the windscreen is down to my stupidity in not turning off the cross-hairs in ThermViewer before making the images.