Why thermal cameras' image quality do not obey their data sheet?

[zajusako]

Hello everyone.
I'm wondering why thermal cameras' specifications do not reflect their image quality.
For examples:
Raytheon BST firefighting cameras have ~100mk NETD, but they produce very good images to my eyes;
Seek thermal have 70mk NETD, but they are barely usable for anything other than a thermal-meter;
Most VOx cameras have 50mk NETD, however they perform wildly different...
Welcome to discuss.

[Vipitis]

optics.

compare the firefighting cameras to the seek. The lens elements are larger, and usually of better coating (to protect against fire, drops, liquids, scratches etc). Also a larger sensor normally means less noise.

those specifications are from the pure sensor and not the camera with sensor in combination with the optics you get. Sometimes the named specs only work when you run your calibration(flag, shutter, subtraction etc...) and wait 5 minutes(probably less) and your sensor is at 300K uniformly.

processing and presentation also make a huge difference.

specs never tell a good story, they are only a guide. non cherry picked or modified example images is what gives you the real answers.

[Bill W]

NETD is a parameter of the detector only, obtained under ideal test conditions for the satisfaction of the detector manufacturer.
They will always strive for lower values and use an f/1 lens.
The quoted values are also generally 'worst case' so a spec '<60mK' might be typically 45mK on test.

What you see in a camera will differ, LOTS.

The following is real if slightly changed to protect the innocent / guilty   

You might change the sensor dynamic range from the supplier test conditions.  Image noise changes, not the NETD.
You might add some multiscanning / averaging after the sensor. Image noise changes, not the NETD.
You might decide that no-one can measure NETD, so write 50mK anyway to avoid losing out on paper tested tenders.
Your display choice will have a huge effect on image perception, but does not change NETD
Your back end processing will have a huge effect on image perception, but does not change NETD.
You might choose the test condition for a great NETD, but be one that is practically useless.  NETD changes, but not the image.
You might alter the NETD in spec to allow selling a range of cameras with the same sensor, and cover it up by adding noise later in the image chain

A couple of practical examples I can quote.
Doing a classical minimum temperature detection test with a camera (MDTD, not MRTD) you get different answers in black hot and white hot, and in different colour palettes.  A fire camera MDTD might be 40mK for an ambient scene but be over 1K when in its' high temperature mode.
MDTD can be higher or lower than NETD.

Probably raised more questions than it answered........

Bill

[IwuzBornanerd]

I don't think I've seen Seek refer to it as NETD, but rather "thermal sensitivity".  That term makes me think of "minimum detectable thermal difference" and I therefore interpret their spec as the one-bit resolution of the sensor.  And even that number varies with temperature along the response curve of the sensor.  In the vicinity of normal room and body temperatures the one-bit resolution of Seeks with their stock lenses may well be around 70mK, but they have 2 bits of noise (4 bits in the early non-pro units), by my reckoning.

Also, as I keep pointing out, in order to properly compare performance you must take into consideration the palette and how it is applied over what temperature range.

[Bill W]

I don't think I've seen Seek refer to it as NETD, but rather "thermal sensitivity".  That term makes me think of "minimum detectable thermal difference" and I therefore interpret their spec as the one-bit resolution of the sensor. 

As suggested above, who knows what 'thermal sensitivity' means to SEEK. Anyone can make up their own figures of merit and give them misleading names.

MDTD however is a defined test of the system  You look at large [1] concentric black bodies and take the temperature difference through the whole cycle of white hot to equal, to black hot and back to white hot.  You get 4 temperatures where the image appears or disappears and average them.  As this is done looking at the image on a screen it is nothing specifically to do with 1 bit of data.

Bill
[1] This means you are not testing resolution at all as you do with the 4 bar chart

[Fraser]

In the early days of thermal imaging manufacturers and customers were pleased with achieving their design and usage goals without focussing too heavily on absolute temperature differential discrimination performance. They just produced the best product that could be achieved with the available technology of the day. I am talking early days though when Liquid Nitrogen was used.

As the thermal imaging technology changed and developed it became available to a wider marketplace and more companies became involved in supplying such equipment. In many cases an OEM would buy-in the parts that were hardest to produce and just place them within their own case and control system. The bought-in parts would have a stated specification and the OEM might choose to simply quote those figures in their final design. As Bill has stated, the OEM may not be achieving such performance in a less than optimal design with lesser quality optics or the displayed image may suffer from poor video processing or graphics overlay stages.

As the thermal camera manufacturer base grew it became challenging for the customer to differentiate between imaging technologies and cameras. Some cameras shared the same basic Raytheon cores so theoretically should have produced very similar imaging, adding to the customers indecision. In order to ‘assist’ the customer the various camera manufacturers employed the magic of a marketing team and slick sales people to convince customers that their particular camera was better than the competition. No greater was this effort than when Microbolometer based Fire fighting cameras appeared in the market in direct competition with the already common BST cored cameras. You only have to search the “Firehouse” forum to see the level of competition that developed between BST and Microbolometer salesmen who frequented that forum in the hope of making contacts and sales. The battle lines were drawn and all manner of “knowledgeable comments” flew with regard to BST and Microbolometer technologies advantages and disadvantages. Some were accurate comments, others were just spiteful and intended to confuse readers.
The battle between BST and the Microbolometer is not the topic here though. However, the salesmen needed ammunition on the ‘battlefield’ and that extended well beyond the BST Vs Microbolometer discussions, it also applied to competition between cameras using the same core technology.

It will come as absolutely no surprise that camera manufacturers marketing teams started to pressure their camera designers to make their products stand out from the crowd and marketing people love gimmicks and impressive sounding specifications. I say impressive sounding deliberately as, like statistics, specifications can be very misleading if the author wishes such ! Just think about those specifications you have seen detailing levels as “dB” rather than the correct dB with respect to a reference as with dBm or dBuV ! Meaningless.

Now marketing teams can get very inventive so it comes as no surprise that glossy brochures were produced that detailed smart ‘new’ capabilities that the oppositions cameras did not have. Some added capabilities were, in fact, useful to the user, whilst others were basically just intended to impress purchasing departments. When you are faced with a swathe of similar looking and specified thermal cameras it is all too easy to start looking for anything different amongst them to give one or two an edge over the others. The customer or purchasing department is unlikely to be an expert in thermal camera design and specification beyond knowing a minimum requirement specification such as Resolution, Frame rate, temperature monitoring capability and NETD. As has already been stated, the NETD specification can be misleading.
Whilst it is true that any purchasing department should arrange for a demonstration of a product to its end users, there is an obvious advantage in either getting through the initial tender paper sift ! You then bring on the big guns in the form of a slick sales person to convince the end user that your product is best or the best value for money of those available. Getting your foot in the door is essential to permit your sales person to ‘perform’ 

So what do marketing teams like to ‘hype’ in the camera specifications ?

My biggest bug bear is hyped resolution which is common amongst Chinese manufacturers. Stating that a thermal camera has a “Resolution” or “Displayed Resolution” of, say, 320 x 240 pixels is meaningless. If I see such in the specifications I always do some digging as that is not a stated OPTICAL or PHYSICAL sensor array resolution. It can be the displayed resolution on the 320 x 240 pixel LCD display after INTERPOLATION. 47x47 pixel Chinese FPA’s are commonly interpolated up to 320 x 240 and the cameras sold as a 240 x 240 pixel unit. The price is usually a dead givaway though ! If it sounds too good to be true, with thermal cameras, it usually is. Always make sure the stated resolution is genuine physical resolution and not some digitally produced resolution using a lower specification source.

Frame rate. Well outside of the USA many thermal cameras have a restricted frame rate in order to make their shipping and sale easier. Higher frame rate cameras are still available outside the USA but the purchase process has a little red tape attached and prices are generally higher as high frame rate cameras tend to sit at the Pro-Sumer and above market level. With a 9fps cap on a camera, there is not much a marketing team can do to gain an edge over the competition but such is not so in the higher frame rate markets where there is no limit on the claims that can be made on frame rate. In truth a high frame rate camera can be run at whatever frame rate the imaging core designer likes, within the physical constraints of the FPA itself. Pixel response time must be considered. Also higher speed ROIC chips and ADC’s on the die may increase self heating which is undesirable. Common high speed frame rates have been 30, 60 and 120 frames per second. A bit like car BHP, more always reads better so a camera operating at 60fps can sound better than one running at a mere 30fps. But wow! What about 120fps ? Well, in truth, the end user needs to consider the intended use of the camera and whether very high frame rates above 30fps are truly necessary. Fire fighters for instance need a camera with a clear easily viewable display of a scene with little motion blur or image refresh lag when the camera is panned. This requirement make a 9fps camera less pleasant to use than a 30fps model. Is 60fps justified though ? Well I leave that to the sales person and practical demonstrations. I personally like to have 60fps available to me but coukd not personally justify 120fps..... but I bet an F1 racing team could easily justify a higher frame rate on their tyre, brake and engine monitoring thermal cameras ! I have noted that the specifications for frame rate on some camera dongles can be misleading. As the dongle is not a complete camera system, it opened a little opportunity for the marketing team to exaggerate. The dongle cameras are totally reliant on their host for the processing and display side of a thermal camera. The host runs an App or program but the performance of that processing and display is heavily dependent upon the processing power of the host. The Marketing team can claim “theoretical” maximum frame rates that are not easily achievable in the real world with a common specification of host. That is to say, theoretically a certain camera dongle can stream data to the host at a rate compatible with the host producing, say 20fps, but only if the host and software are up to the task ! In a real world scenario with, say, last years model of mobile phone, and a less than perfect application that is also sharing the host processors power with other housekeeping and apps, the frame rate drops to a mere 10fps. Is the original 20fps specification a lie ? Nope.... it is the best that can be achieved and not a stated nominal. A nominal FPS is hard to specify as the camera dongle is a slave to a host over which the camera manufacturer has no control. This is the nightmare of using an uncontrolled host for the processing rather than a closed loop system of known specification, performance and stability. With phone dongles the camera manufacturer has all manner of ways to avoid being held to account for “ambitious” specifications.

NETD : this has already been covered by Bill. NETD is just a baseline. It is not the specification of the whole system with respect to its ability to DISPLAY or produce data for a stated minimum temperature differential.
Rubbish video processing stages can destroy the systems minimum temperature differentiation figures or if the display firmware and electronics are incapable of displaying a claimed minimum temperature differential, the specification is pretty meaningless. It must always be borne in mind that the marketing team desire the best possible specifications to come out of their in-house product testing. They care not whether an end user will actually be able to see any difference in performance against the competition when the camera is in normal use. Lab testing can be very different to real world use...... just think of the difference between using a thermal camera in a nice clean lab with precision Black Body sources, and using the same camera inside a very hot smoke filled room with fire licking around you, a dirty and wet Breathing Apperatus faceplate and soot covering everything including the camera lens! A fireman is less concerned about the minuti of NETD specifications and is more interested in a camera clearly displaying either a casualty amongst the mayhem of a fire or a door that is so hot that he knows there is a fierce fire beyond it ! A well designed ergonomic Fire fighting camera with decent, if not exceptional performance may well be preferable to a lab grade camera with amazing performance but that is unwieldy and far from optimum for the task at hand. This is a specific case scenario I know, but the principle applies to most cases. A thermal camera needs to be so much more than just impressive specifications. It must suit the intended task and its design should be ergonomic and user friendly. I remember seeing very capable test equipment sitting on a shelf once and asked why it was not used, yet lesser equipments of similar usage were. The answer was simple “no one likes to use that XYZ unit because it is a nightmare to drive and easy to configure incorrectly, so producing poor results”. A lesson there for us all me thinks.

Well that is enough from me for now. I have written more than intended !

Writing this on an iPad so it is likely full of typos etc.... I will correct them later

Fraser

[Fraser]

As a side note, I have just ordered a book that discusses the specifications and assessment of thermal imaging systems. It is called “Thermal Imaging Cameras - characteristics and performance” by Williams.
I have resisted purchasing the book to date as it covers a narrow (but important) topic and is quite expensive. I found an excellent used copy for £35 so decided to add it to my library. It is a book likely aimed at thermal camera system designers, scientists and those who write technical tender invitations but I am sure I will find it’s contents of interest.

https://www.crcpress.com/Thermal-Imaging-Cameras-Characteristics-and-Performance/Williams/p/book/9781420071856

Fraser

TOC follows.......

Table of Contents
Preface: Thermal Imaging Cameras

Biography

Symbols and Abbreviations

An Introduction to Thermal Imaging

Aim of This Book

What Is Thermal Imaging?

Atmospheric Transmission

Choice of 3–5 µm or 8–12 µm

Important Factors in the Application of Thermal Imaging

Specifying and Measuring the Performance of a Thermal Imager

..............

Thermal Imaging Cameras and Their Component Parts

A Basic Thermal Imager

Image-Forming Optical System

Windows for Thermal Imaging

Scanning Mechanisms

Radiation Detectors

Cooling Detectors

Special Signal Processing and Other Requirements

Displays

Computer and Software

References

..............

Industrial and Commercial Thermal Imagers and the Facilities

They Provide

Introduction

Thermal Imagers That Measure Actual Temperature

Software Facilities for Industrial Applications

Thermal Imagers for Surveillance Types of Application

Imagers Used by Firefighters

Manufacturers’ Data Sheets

References

.................

Performance Parameters for Components of a Thermal Imager

Introduction

Lenses

Detectors

Scanners

Displays

References

..............

Performance Parameters for a Complete Thermal Imager

Introduction

Factors Affecting Performance

Performance Parameters

Frame Rates and Readout Timings

Other Factors

References

.................

Basic Equipment for Testing and Calibrating at Thermal Wavelengths

Introduction

Radiation Sources

Infrared Detectors

Cooling Detectors

Optical Systems and Collimators

Integrating Spheres

Spectral Filters

References

....................

Measurement Procedures and Techniques for the Principal

Components that Make Up a Thermal Imager

Introduction

General Measurement Procedures

Lenses and Optical Systems (see also Section 4.2)

Detectors and Detector Arrays

Scanners

Displays

References

..............

Measurement Techniques and Procedures for Complete

Thermal Imagers

Introduction

MTF

Nyquist Limit

Aliasing

Signal Transfer Function (SiTF)

Noise-Equivalent Temperature Difference (NETD)

Minimum Resolvable Temperature Difference (MRTD)

Minimum Detectable Temperature Difference (MDTD)

Objective MRTD

Objective MDTD

Slit Response Function (SRF)

Aperture Response Function (ARF)

Temperature Measurement Performance Characteristics

Emissivity Range

Narcissus

Veiling Glare Index (VGI) and Uniformity Index (VGUI)

Scene Influence Factor (SIF)

Field of View (FOV)

Close Focus Distance

Spectral Response

References

...............

Calibration and Alignment of Test Facilities

Introduction

Calibrating and Aligning a Collimator

Radiometric Characteristics

References

................

Applications of Thermal Imaging Cameras

Introduction

Industrial Applications

Advanced and Specialist Applications

Medical and Biological Applications

Military and Civil Surveillance and Other

Applications

References

..................

Appendix A: Objective Measurement of MRTD and MDTD

Performance Parameters That Determine MRTD and MDTD

Theory for MRTD Measurement

Theory for MDTD Measurement

References

.............

Appendix B: Sampled Imaging Systems and Aliasing

What Are Sampled Imaging Systems?

Image Formation by Sampled Imaging System

Basic Theory

Measures of Aliasing

MTF Measurements Techniques

References

.........


End

[eKretz]

That looks like a very good book Fraser. Thank you for pointing it out to us. I've just found a secondhand copy for $39 USD.

[Bill W]

Fire fighters for instance need a camera with a clear easily viewable display of a scene with little motion blur or image refresh lag when the camera is panned. This requirement make a 9fps camera less pleasant to use than a 30fps model. Is 60fps justified though ?

It should also be noted that some '9Hz' cameras are indeed 9Hz, while others may be 8.3 or 7.5Hz as that makes the dividing easier (1 new frame in 4 gets displayed). 
The human eye-brain works around 10Hz so sees a huge difference between them, it is almost taken in with a strict 9 images per second.

Some also think a 4 frame average makes a <9Hz camera, that is bad motion blur especially if it is a classical digital average.

Bill

[Bill W]

Here's an example of makeupaspec....

Without the help of Google, what do you think "Highest ISDR of any fire service TIC"  "Total ISDR of 4701" means, and how would you calculate it  ?

ISDR = Instantaneous Scene Dynamic Range.

Bill

[Fraser]

Without the help of Google.....

I would think that ISDR was similar to ADU dynamic range in an ADC based system.

To measure instantaeous Dynamic Range I would likely wish to use two variable temperature black bodies that would be set at temperatures spaced apart to detect when the ADC output had reached its limits or the system connected to the ADC was no longer capable of displaying change. The camera would need to be set in a manual configuration mode to ensure stability of the test. All that the test would be showing is the maximum useable range of the microbolomter and ADC combined with the display systems ability to deal with such in a way that a user can analyse.

Now to the number “4701” ...... this would appear to be a nonsense as it has no reference units to indicate whether it is ADU, a unit of measure or a ratio. 4701 can only mean something when placed in an appropriate context and when that context is clearly detailed by an SI unit or explanatory note.

The nearest I can think of to ISDR in daily use would be contrast ratio of a display. If no change can be seen by moving the drive signal further into the dark and lighter areas of a display, the maximum instantaneous contrast ratio has been reached. LCD displays are often quite low contrast ratio at around 1000:1. This is important if specifying an effective ISDR as 4701:1, if the LCD display is only capable of 1000:1. Such is normally dealt with by presenting only a section or ‘window’ of the ADC’s full dynamic range to the display. This windowing effectively limits how much of the claimed ISDR is available to the user at any given time though.

Fraser

[Bill W]

Without giving much away, I can confirm ISDR is dimensionless - in fact that makes it even more bizarre.

Bill

[Vipitis]

So this thread has accumulated some nice information, which I haven't yet read. Frasers bookfind looks interesting, but I guess it's more on the technical and historical side and not on the creative direction with fancy digital processing I am after.

ISDR from the name given, I would think of being some kind of metric to measure the steps of difference when you initially turn the camera on. But it implies a about 12bit signal giving you even a little more. I would believe that it's made up to be a strange combination of usually used numbers, like NETD and boot up time. Or even a something unrelated to the sensor and core like the display or palette.

But please, let us guess for a few more days before you reveal it.

[Fraser]

The book by Williams on thermal camera performance and characteristics arrived today. A quick skim through it reveals an interesting reference that explains the many different characteristics that can be profiled for any given thermal camera system and their effects on camera performance. It is a larger topic than I had expected. Basic thermography theory and camera design is covered to ensure the reader is aware of the context of the characteristics and testing.

I am pleased to have procured this book but it is not what would be called casual reading or an essential text for thermal camera users. Perfect for my interests in the technology though There are other books more suited to thermographers and serious users of thermal imaging equipment. (one being Infrared Thermal Imaging by Vollmer and Mollman)

The sections on testing a thermal cameras performance and the equipment required were very insightful 

Fraser

[Fraser]

@Bill W,

So Bill, please put us out of our misery and tell us of the mystical ISDR and the meaning of the 4701, if any.

From what you have already said, I am expecting this ‘specification’ to be some mangled derivative of a genuine Dynamic Range specification that Marketing thought sounded superior to other manufacturers capabilities. I assume the sales persons were briefed on it so they could explain the meaning when knowledgeable people asked what it meant.

I am assuming that the company that stated ISDR is trying to say that their camera is very sensitive, whilst also able to cope with very large temperature differentials within the scene. Sort of like the standard spec found on most fire fighting cameras really 

Fraser

[Bill W]

OK the answer is the attached datasheet from MSA.

ISDR, Instantaneous Scene Dynamic Range is obtained by dividing the available range by the NETD, but you are allowed to add them up if it has more than one range !

*Instantaneous Scene Dynamic Range is the
combined value of a sensor’s sensitivity and
temperature range to determine the number
of temperature differences that can be at the
scene at any given instant.

The odd number comes about because the ranges are in Fahrenheit and the NETD in mK

It is unusual (but typical of Indigo, the then core supplier) to own up to the NETD in a low responsivity range, in this case 65mK and 300mK

So 0°F to 320°F at 65mK (2735 ISDR) + 320°F to 1060°F at 300mK (1966 ISDR) = 4701 ISDR

A contemporary Argus 4 (HR320) had 3 modes, but only quoted MDTD
-50°C to 150°C at 60mK
150°C to 450°C at 100mK
450°C to 1000°C at 300mK
ISDR = 8167

Oh how we laughed when we did that sum and found out it was more than the MSA.....

Bill

[Fraser]

 

 

[Vipitis]

I see this spec being bullshit and awfully calculated. But I get the idea.

If you do it dynamically you may end up somewhere and then decide by the range. It would give you the average sensitivity across the whole range.

To calculate it you would need a function that is NETD(K) in mK and than calculate it's area to get ISDR.

[Bill W]

I see this spec being bullshit and awfully calculated.

Yes, that was how it was done at the time, as Fraser has noted from the Firehouse forum bunfights

To calculate it you would need a function that is NETD(K) in mK and than calculate it's area to get ISDR.

That can also be a variable across the scene if you have non-linear image processing going on such as equalisation.