Problems with RAW 14bit image in FLIR One

[Ben321]

By pushing it to its upper limit by pointing it at the heating element on an electric stove and taking a picture using raw 14bit mode in the SDK sample app (which I'd modified to be able to save the full 2-byte-per-pixel image data), I discovered 2 problems with the raw image, that indicate it is in fact NOT truly raw:

1) The hottest part that should saturate the sensor, and be constantly at the maximum value, instead wraps around back to 0 making the hottest part of the image look like it's actually colder than the coldest part of the image.

2) The largest number I found the image should have been 16383, the maximum value that can be stored in 14 bits. But it wasn't. The maximum value I found in the image was 65535, which is the largest value that can be stored in 16 bits. Rather than simply having the 14bit data be alligned to 16 bits by having 2 unused bits, it seems that they chose to scale the 14 bit data to 16 bits (calculated by outputvalue=inputvalue*65535/16383).

Why not call this high-bit-depth data, instead of raw data? It's not raw data. Raw means "unaltered", not "mostly unaltered". Even worse, they are very specific in how many bits are used in the name of the constant that defines this image type. The official name for the constant in the FLIR One SDK that defines this image type is "ThermalLinearFlux14BitImage". That is a lie. It seems like they are trying to make the users of the SDK think that the image is more raw than it is. It is NOT a 14 bit image. It is a 16 bit image, that got scaled to 16 bits.

Is it really that hard for them to create an SDK function that exposes the TRULY RAW 14 bits per pixel sensor data?
Or does this conversion take place in the thermal imager itself, via the firmware running on the device's microcontroller?

[tomas123]

Do you know, that all sensor values in radiometric mode are not really RAW values, got from ADC!

Also I wrote in your thread, that the raw values send over USB divided by factor 4:

I found a simple solution to check the factor 4 between "Lepton RAW" and "Flir radiometric jpg RAW":

The tea bag simulator of Flir SDK use real Lepton sensor values. You found it in the SDK file sampleframes.zip (6MB).
I made with the SDK App a shot from the (random) tea bag frame 00051.
Thereby we got a Flir radiometric jpg with an embedded calibrated RawThermalImage with a good temperature range from min/max=34°C/65°C
(the unrealistic min temperature is a result of bad calibration values for the simulator)

first some image magick steps with the lepton sensor values of frame 00051 (file 00051-lep) from SDK sampleframes.zip
("-chop 2x0+0+0 -chop 2x0+80+0" removes the 4 additional vertical lines with extra unknown informations)

Code: [Select]

// convert sample frame 00051 from SDK
> convert -depth 16 -size 164x120 gray:00051-lep gray:- | convert -depth 16 -endian lsb -size 164x120 gray:- -chop 2x0+0+0 -chop 2x0+80+0 -rotate 90 00051-lep.png
// multiply with factor 4
> convert 00051-lep.png -fx "4*u" 00051-lep4x.png
// get channel statistics
> identify -verbose 00051-lep4x.png
  Channel statistics:
    Pixels: 19200
    Gray:
      min: 14220 (0.216983)
      max: 20840 (0.317998)
      mean: 15014.9 (0.229113)
      standard deviation: 1218.95 (0.0186)
      kurtosis: 5.54453
      skewness: 2.30479
and now compare the statistics with the "real" Flir radiometric image from the SDK app

Code: [Select]

// extract RawThermalImage from a Flir radiometric image
> exiftool -b -RawThermalImage FLIROne-2016-01-18-10-33-07+0100.jpg > 00051exif.png
// image is postprocessed and resized to 320x240
> convert 00051exif.png -resize 160x FlirAppRAW120x160.png
// get channel statistics
>identify -verbose FlirAppRAW120x160.png
...
  Channel statistics:
    Pixels: 19200
    Gray:
      min: 14266 (0.217685)
      max: 20831 (0.317861)
      mean: 15021.4 (0.229212)
      standard deviation: 1221.27 (0.0186354)
      kurtosis: 5.44004
      skewness: 2.28451
By deducting the Flir post processing steps to the original RAW values (see patterns below) we got a great result!
A sample:

max-min=65-34=31°C
max-min=20831-14266=6565 digits
-> 1 digit = 31/6565=0.0047 Kelvin
The difference between mean in this images is:
(15021.4-15014.9)*0.0047 Kelvin= 0.031 Kelvin 

You also see, that the last digit ADC "resolution" of Lepton in this range is > 0.0047*4 = 19mK (useless because of strong noise, see second image below with 4 Kelvin scale)

here is a real live sample from Flir One G2 (shot after a small warm up time of about 2 minutes):

I saved with the SDK.app simultaneously a upscaled Flir Radiometric JPG  and a real Lepton ThermalLinearFlux14BitImage.

Afterwards I rebuild with my old panorama script (see my footer) a real size 160x120 Lepton radiometric jpg (a Flir format).
You can load this sample jpg images in Flir Tools and compare the quality.

First a original image shot with the Flir App.
The App crop    the Lepton sensor to about 120x90 Pixel.
Please note the artefacts/patterns!
Flir makes a nice lens distortion correction of the Lepton sensor for best MSX overlaying 




real  Lepton sensor 160x120 (no image postprocessing and with noise/grain because the temperature spread is only 4 Kelvin)

[Bill W]

Told you !
https://www.eevblog.com/forum/thermal-imaging/another-smartphone-thermal-imager-from-i3/msg926744/#msg926744

So FLIR's use of the term 'RAW' offends ?  It seems in line with 'RAW' as used on visual cameras, see
https://en.wikipedia.org/wiki/Raw_image_format
ie inherent sensor nonuniformity and dead pixels are removed before saving.  The maths to do this will go to 32 bits and be truncated back to 16. 

Do you get all the possible LSB values ( 0,1,2,3,4,5,6,......255) or simply two trailing zeros instead of two leading ones (eg 0,4,8,12,16,20...252 )?

Bill

www.fire-tics.co.uk

[Ben321]

Do you know, that all sensor values in radiometric mode are not really RAW values, got from ADC!

I'm not talking about radiometric mode (what FLIR One SDK calls ThermalRadiometricKelvinImage). I'm talking about what is SUPPOSED TO BE a true raw image (what FLIR One SDK calls ThermalLinearFlux14BitImage). That is supposed to be a completely raw, uncalibrated signal, straight from the microbolometer to your Android phone or tablet, with the intent that the app designer using that mode will write their own calibration routine (calibration done in software rather than in hardware). But FLIR's name for it (ThermalLinearFlux14BitImage) is a lie, because it is NOT 14 bits. It's stretched to fill the entire 16bit range possible using 2-bytes-per-pixel. And even worse, an overflow causes it to wrap around to 0, instead of simply saturating at 65535. I consider this to be a flaw in their SDK, and hope it is corrected in future versions of their SDK.

Or maybe, somebody will figure out how to alter the FLIROneSDK.aar file itself, and implement true raw image as a feature in their hacked version of the SDK. Since AAR files are just zip files, it would be easy to extract the Java classes from it, then a good hacker should be able to disassemble the .class files into .java source code files, edit these files in the way they handle the FLIR One USB protocol, so that they will attempt to actually get true raw images from the FLIR One, then recompile the .java files to .class files, and finally put these java class files back into the AAR file to replace the defective ones that FLIR created.

[tomas123]

Believe me, I hacked since years some steps of image processing of the Flir cameras.

The noticeable market edge of Flir is the post processing of the sensor values.
The true sensor raw is a useless signal for you.

The microbolometer generates some noise and drifts, which are filtered with the flat-field correction (FFC):
Pixel Noise
Row Noise
Column Noise
Uniformity

The next step is the bad pixel mapping.

After then the Flir firmware converts (in radiometry mode) this post processed signal to a value which is linear to radiance [W / m^2 (per nm)].
This computational result is the so-called 14 Bit Raw signal (like ThermalLinearFlux14BitImage)

If you are interested in such things, then you can download interesting knowledge papers and sensor manuals from the Flir Web Site.
A good entry point is the datasheet of die Lepton 3 sensor
https://www.eevblog.com/forum/thermal-imaging/flir-one-thermal-imaging-camera-teardown-and-hacks/?action=dlattach;attach=206832

I have a Flir Exx. There is a service menu, where I can deactivate some postpressing features.
The result is "impressive".

If you are interested in the image quality of really raw microbolometer values, then read the seek thread in this forum..

[frenky]

Comparison like this makes me wanna smash my Seek with a hammer and just move on.... 

[Fraser]

Frenk,

I do feel for you but I have been very impressed with what you have achieved with the SEEK camera. Sadly my initial thoughts about the SEEK Thermal products have been accurate and sadly it is not a paltform that I feel I can recommend to anyone.

To be totally fair regarding the above sample images. The E4+ is better perfoming than FLIR ever intended or would want. What was achieved with the E4 is truly great and offers great value for money to those who buy it and upgrade the performance. The FLIR ONE Gen2 and its little LEPTON 3 will struggle to produce images as good as the E4+. Some clever processing provides a decent image but the noise levels are still quite high. The applications that are being developed for the F1G2 by members of this forum impress though. Super resolution mode would be a very interesting capability 

With regard to the F1G2 truly raw data from the microbolometer ROIC, I suspect the image will be almost as noisy and ugly as that produced by the SEEK camera. it is clever video processing developed over several decades that has equipped FLIR to make a pretty ugly image look acceptable. Other thermal camera manufacturers have had to develop their own noise processing algorithms, or rely upon a 3rd party cores that contain all the image processing needed, and may be married up to their camera chassis and electronics. For this reason I believe FLIR would offer what they call RAW (but processed) output for further manipulation by an OEM as required, but they would see little point in providing the RAW ADC levels as those are likely very challenging to use without some serious background in the issues involved and noise processing in thermal cameras. Whilst Ben321 might have such skills, I doubt many other customers would want to do such work and so would rely on FLIR's skills and reputation for image processing instead. IN my thinking, FLIR's term 'RAW' means a data output that may be adapted to a users needs with range selection, overlays and measurements possible.

Fraser

[Ben321]

After then the Flir firmware converts (in radiometry mode) this post processed signal to a value which is linear to radiance [W / m^2 (per nm)].
This computational result is the so-called 14 Bit Raw signal (like ThermalLinearFlux14BitImage)

Then it should be called a 16-bit post processed signal. This is because the processing produces a number that makes use of all 16 bits. The name of the constant for that image type in the SDK should then be "RadiantEnergy16BitImage". And I think I know why FLIR calls it 14 bits instead of 16. It's because their documentation on the Lepton chip itself specifies that the sensor uses 14 bits, so by having an image type name with the number 14 in it, it implies (albeit incorrectly) that the values you are looking at are raw values directly from the Lepton chip. This decieves the end-user into believing they are looking at raw values, and that there is an unlimited number of possibilities they can do with it (which would be true if it really was raw, but sadly it's not). This tricks the end user into being complacent with the product they have, never realizing what they are missing out on, if only they had access to the true raw data from the chip. So the customer is a happy customer (even if he shouldn't be), and and that makes FLIR happy when the customer is happy, though the customer would definitely not be happy if they knew just how much they were being deceived. And trust me, I'm a not very happy customer, now that I know how much I've been deceived, thanks to all you guys' help here on these forums, revealing the truth about how FLIR hides the true raw values from not only end-users, but also from SDK users (such as myself) who want to develop apps for end users.

[tomas123]

Hi Ben321,

you are a hard nut!

Another sample:

On my Canon Cameras the Camera RAW file format is also a postprocessed image of the Bayer filter.
There are some communtiy projects like Magic Lantern and CHDK "Canon Hack Development Kit" which has some tricks to get the really RAW datas from Bayer filter, but about this are only a infinitesimal part of Canon user interested in.

back to Flir:
Flir never said, that RAW is the result of the ADC with microbolometer sensor voltage as input.
The Flir knowledge papers described RAW "as linear to Radiance with an offset".
The Flir datasheets explains detailed how they get this Radiance-RAW (uniform image etc.).

[Ben321]

Hi Ben321,

you are a hard nut!

Another sample:

On my Canon Cameras the Camera RAW file format is also a postprocessed image of the Bayer filter.
There are some communtiy projects like Magic Lantern and CHDK "Canon Hack Development Kit" which has some tricks to get the really RAW datas from Bayer filter, but about this are only a infinitesimal part of Canon user interested in.

back to Flir:
Flir never said, that RAW is the result of the ADC with microbolometer sensor voltage as input.
The Flir knowledge papers described RAW "as linear to Radiance with an offset".
The Flir datasheets explains detailed how they get this Radiance-RAW (uniform image etc.).

Then it still shouldn't be called ThermalLinearFlux14BitImage. 14-bit image refers to the number of bits-per-pixel in the true raw image that comes directly from the sensor. The processed image that is supposed "raw" image uses all 16 bits, so it should instead be called "ThermalLinearFlux16BitImage". But I know why they don't name it correctly. They do that, because they know that anybody who knows the specs of the Lepton chip will know it has 14 bit output, and by calling the raw format ThermalLinearFlux14BitImage, they falsely lead the person using it into believing that they are accessing the true output from the sensor itself. This gives the person a sense that they can do an unlimited number of things with it, because they will think it's truly raw, but really they will be significantly limited, because it's actually already been processed. It's this kind of dishonesty that I don't like.

[Redshift1340]

I think I hear an echo

[Ben321]

Ok, I think I've figured out more about how this "raw" image is calculated now.

It includes the shutter "dark frame" correction. I can tell this, because when I took a picture of a hot object (my toast being made in the toaster this morning) it left a brighter than average spot on the sensor where the image of the heating element was (apparantly the IR radiation was so strong that it actually caused localized heating of part of the microbolometer array in the Lepton chip in the FLIR One). However, after manually sending the signal to do a shutter correction, that bright afterimage disappeared (was able to be calibrated out of the image), so obviously this isn't the true raw image (which would have no corrections applied). This effect was just temporary though, as after the localized heating started to cool down to the correct temperature, the shutter corrected region actually started to look darker, so I had to do another shutter correction once the localized heating effect was completely gone, in order to now remove the darker region

A can also now tell you why it looked like it wrapped around to 0. The truth is it didn't actually wrap around to 0. If it had have done so, some of the pixel values in the center of this "sensor overload" region would have been just above 0, where it was at its absolute hottest. But that's not what I observed. When I looked at the raw data, it instead had ALL of the overload region EXACTLY equal to 0. This suggests that 0 is actually used as an error indicator value, to signal a sensor overload. If 0 is used as an error value, then that means it can't be a valid actual signal value (no possible valid sensor value, either hot or cold, will produce a value of 0 in the raw data). This means that there must be some additive offset applied to the raw data, to make it so that 0 IR radiation intensity (the amount of IR radiation emitted by an object with a temperature of 0 Kelvins, absolute zero) does not actually produce a value of 0 in the raw data. Instead it would be stored as a value just above 0, so that the value 0 is reserved as an error value.

Now that I know this, it should be pretty easy to correct for this error indicator value, and bring it back to showing the actual saturated sensor value, by writing a program that takes this raw data, and simply fills in all pixels that have a 0 value with the value 65535 instead. Though I haven't tested it yet, my one concern might be that values that are "too cold" might also be indicated by a sensor overload error (raw value = 0) instead of indicating the actual sensor value. In that case, it would be impossible to tell if an error was caused by it being too strong of an IR signal or too weak of one. Hopefully, if there is a cold overload value, it will be indicated by a different error indicator value (maybe 1, instead of 0), so that it could be corrected for as well. The only remaining question then is what is the actual lowest raw IR sensor value supposed to be?


PS:
It should still, as I said before, actually be called ThermalLinearFlux16BitImage, instead of ThermalLinearFlux14BitImage. This is because all 16 bits are used to store this processed copy of the raw data. It is dishonest on FLIR's part to call it ThermalLinearFlux14BitImage, as this gives the false impression that it's raw-er than it actually is. It gives the impression that you are accessing the 14 bit raw data directly from the sensor, without processing, when you are in fact not doing that. Sadly, I think that is the impression that FLIR was trying to give, intentionally to deceive users of the SDK, so that they would think they were getting a purer, more raw form of the data than they actually are. I don't like companies that use dishonest business practices.

[tomas123]

This are old news.

I wrote above something about 14 bit raw and FFC.

This overload pixels you  also see in raw images of Flir Exx

[Ben321]

I just realized I made a mistake in my most recent assessment. The 0 value is NOT an overflow indicator. Instead what I had originally thought was happening is actually happening. It is going above 65535 and wrapping around back to 0, and continuing up from there. I can tell that because I got a better picture to work with now, and used the levels control in Photoshop several times consecutively (dragging the highlights slider all the way to the left) to boost the brightness dramatically (making even the darkest, most black-like, pixels have a value in the visible gray range) while in 16bits per pixel mode. This has allowed me to confirm my original suspicion, that there is an overflow occurring that goes above the value 65535.

However, there may be a dedicated "overflow range" down near the bottom of the brightness scale, that is below the value that would indicate no detected IR radiation, in order to indicate that there is an overflow on the upper end of the scale. If this is the case, then that still means that an offset is being added to the detector's raw values. Unfortunately I don't know how to determine what this offset is, as the only accurate way to do this would be to point it at an object that is at a temperature of absolute zero (or within a few millionth's of a degree above absolute zero, like is found in some very specialized physics experiments). If I'm going to detect overflows, I'm going to need to be able to find the range of values dedicated to indicating an overflow, and to do this I will need to know what the raw value offset for absolute zero IR radiation is. Can somebody help me out here with this?