Super-Resolution enhancement - why do we not see more of this ?

[Fraser]

For those unaware there have been methods used in the past to increase the effective resolution of a thermal camera FPA to  create far higher resolution images than could be achieved with the FPA at its native optical resolution. These methods have included using a spinning disk containing germanium prisms that offset the image presented to the FPA pixels and software solutions that use the natural movement of the thermal camera when hand held to capture multiple images for merging in software. The spinning prisms ide is expensive and complex to implement and soon passed into thermal imaging history. The software approach is a far more practical option, especially with the power of modern processors that could easily process the imagery in real time. Note that we are not talking about interpolation or clever upscaling techniques that just use the limited raw resolution of the FPA. We are talking about the combining of multiple sets of fresh pixel data to create a true higher resolution image.

Super resolution has been available in recent years in two main forms…. On-camera dynamic super resolution of the live image and off-camera post processing using the super resolution imaging data from the cameras saved images. Again, this is not interpolation of data, it is combining of unique data sets. I own a Agilent thermal camera that provides Super-Resolution and Testo also offer it in some of their cameras. It is the on-camera dynamic super resolution that most interests me as it is seen whilst actually using the camera. Whilst it is true that such a technology could lower the true frame rate of a low powered thermal camera, if a decent image processor is used, it would not impact the frame rate significantly as the microbolometer may be read at higher speeds, such a 120fps whilst the images are presented to the user at 25fps, or possible faster.

So why are we not seeing more of this software based resolution enhancement on cameras with the, now common, 256 x 192 pixel FPA’s ?

Note that the tripod mounting of a camera with software based super resolution defeats the basic working principle of the system and n image resolution enhancement is to be expected.

Fraser

[Fraser]

A comment on the Agilent U5855A thermal camera that I own when it was first released. True optical 320 x 240 pixel resolution from a 160 x 120 pixel microbolometer 

“ In predictive maintenance, the U5855A TrueIR thermal imager allows engineers to safely and efficiently identify potential faults without shutting down the systems or disrupting the productivity of an industrial plant. The U5855A comes with Fine Resolution capability, which enhances the quality of thermal images by reconstructing the image based on multiple continuously captured infrared frames. This allows the U5855A to achieve an effective image resolution of 320 x 240 pixels from a 160 x 120 pixel detector. With this feature, the U5855A provides four times more resolution than typical 160 x 120 thermal imagers. Together with a 4x digital zoom, the U5855A reveals finer details, especially when measuring from a distance away.”

[Fraser]

The brochure for the Agilent U5855A camera details how dynamic resolution enhancement works.

https://www.richmondscientific.com/wp-content/uploads/2022/05/Keysight-TrueIR-U5855A-Brochure.pdf

Fraser

[tszaboo]

They are probably using the visible camera to detect movement. Most low cost cameras don't have this sensor fusion, and the low resolution image is  not really usable to detect motion. That's just my guess. I don't know why they don't have it, a CMOS camera is very cheap nowadays.

[DaJMasta]

That camera was originally an NEC design, could it be proprietary/patented?  I imagine it does take some extra grunt in terms of FPGA, but compared to the cost of the device (at least for the older, pricier cores) it's a pretty small amount of the total.

And you would know better than I, but could the exposure time actually be a limiting factor?  It could be that the newer budget detectors are too noisy without longer exposure times, limiting the frame rate with that rather than readout speed (or export treaty...).

It could also be that the raw resolution is sufficient to be competitive and for whatever reason they don't want to advertise a resolution near the highest easily available by upscaling their higher resolution cores.

I'm not much of an image processing guy, but maybe it's just harder to pull off well than it seems?

[CatalinaWOW]

I suspect that it is a combination of factors, all coming back to cost and proposed market.  In principal you can find the motion of the camera without another sensor, but it takes a lot of horsepower to do it to the resolution required and can introduce a sizeable lag in the image output.  The method basically does a best fit estimate of the camera motion based on an assumption that the scene doesn't change over the processing interval and some sort of mass-mounting model for the camera.  This assumption of scene invariance gets worse as the frame sequence gets longer, while the quality of the estimate improves with length of the sequence if the assumption is true.   The motion model of the camera is also kind of tough to get right in a camera that is sold commercially.  Who knows what kind of mount the camera will be on, or what the environment will be.  Then given that you know where the camera is pointing the resolution improves proportional to the square root of the number of frames processed.  Under ideal conditions you can double resolution with four frames, reality is somewhere below that.  The simple thought experiment of motion in one plane only which results in no improvement in the cross plane illustrates why more frames are often required.

So it would appear that those making these cameras have determined that a multi-frame output lag and the need for significant additional memory and processing resources aren't making their products more marketable and therefore haven't put the effort into these implementations.

Of course those who don't need real time results can put their camera on a mount that does a small repetitive scan (on the order of 1 pixel) and completing the scan after a half dozen up to a couple of dozen frames and then post process the resulting frame stream.  I have seen this done a few different times and it works reasonably well.

[nikitasius]

From amazon.fr this morning, 238€ offer
Hikmicro ECO
96x96 in 240x240

They call it "SuperIR".


Upd: btw case looks like Hikvision. If i remember well Fraser told that there are no firmware for DS-2TP31 but only for 31B. Maybe there are Hikmicro clone of 31 w/ firmware somewhere?

[Fraser]

Hi Nikitasius,,

Yes I saw those ECO cameras advertised. I was surprised to see such a low FPA resolution in the camera. I have not looked into how they actually create the 240 x 240 pixel image from those 96 x 96 pixels. Interpolation will likely be involved and Super Resolution techniques do look to be present as well.

I think I now have the firmware for the DS-TP31 as it is also known as the E1L. It is a strange situation with regard to model identities. Hikvision and HikMicro are the same company and the Hikvision DS-2TP21B and DS-2TP31B are both available from HikMicro in their standard, non fever detection, configuration as a general use cameras. HikMicro use a different naming convention though. DS-2TP31 is the E1L and DS-2TP21 is the M10. Firmware is available on the HikMicro site for both.

https://www.hikmicrotech.com/en/support/download-center/firmware-download/

Fraser

[nikitasius]

Well, just installed E1L

There are changes in a menu, but range still locked

[nikitasius]

Btw regarding this URL FW for 2TP 31 is E2L
http://ftp.hikvision.ru/01.%20Firmware/01.%20Hikvision/10.%20Thermal/1.%20Firmware/00%20Temperature%20Screening%20Cameras/2TP%20Handheld%20Series/TP31/

But it's failed to install. Only E1L goes well.

[nikitasius]

Btw, this video:

Flir i7, Flir E4 & Flir E8 (hacked E4 guess).

Flir i7 is 140x140
Flir E4 is "80x60"
Flir "E8" is 320x240

Also as it's hacked version, normally no more fake noise issues.
I'm surprised how good picture from i7 looks. Pretty clear, contrast & detailed.
Actually Flir i3 cost cheaper than E4 (used items market, sure) and it could be hacked to i7 (sure it doesn't have MSX but thats not a big deal).
Are Flir seriosly enhancing low-res images from i7?
Is i3/i5/i7 have better bolometer?
Upd: found it https://www.eevblog.com/forum/thermal-imaging/why-e6-costs-3x-more-than-tg267-(theyre-basically-same-camera)/msg3235738/#msg3235738 it have bigger pixels, while it have not the best one (compared to TAU2) videoprocessing 🤔
Also i3 was released in feb'2012, i7 in jan'2013, E4 in feb'2017 (datas from amazon.com). So E4 should be "more modern".

[nikitasius]

I'm not sure i3 is still hackable.

hmm.. new i3 not hackable? i've seen threads on forum about, an example https://www.eevblog.com/forum/thermal-imaging/flir-i3-sensor-to-120x120-pixels-unlock/

but it just hacks 60x60 to 120x120, while i7 seems to have 140x140. But i've seen in another threads folks asking for 140_140 bin files. So.. in theory maybe it could be hacked from 120x120 to 140x140.
Fraser wrote somewhere if my memory aren't fails that it's FPGA limitation, so we can't get real full res, but 140x140 also seems cool + if you correct and they do image downsampling the 140x140 could absolute max we can have.

[Marco]

Modern "normal" cameras have mechanical image stabilization and are made in high enough volume that it only adds a couple bucks to the BOM. So they can stabilize each subframe and then jitter the sampling position. It's an essential part to do superresolution right. You can't really recover the aliased image content after it has been smeared by camera shake.

Somehow I doubt cheap infrared cameras have mechanical image stabilization on the thermal part.