Application of sparse L1 pursuits to resolution enhancement.

[coppice]

The whole idea that you can discard most of the data and get a better image boggles my mind.

Because you can't?

You have to consider information theory and observability. You cannot create information out of nothing, or extract information that is not present in the source data. What you can do is interpolate, and create the illusion of data that was not there to begin with.

This whole subject smacks of free energy and perpetual motion. Only it is wrapped up in so many words and so much mathematical jargon that the empty hole in the middle cannot be seen.

There is real mathematics and solid theory, but it embodies a truth that is much more mundane and much less miraculous than some of the amazing words would make it appear.

You seem to be confusing data with information. In most situations you can throw away lots of data and lose no information. That's what lossless compression is all about. In things like images you can carefully throw away a huge amount of data and lose little information. That's what lossy compression is all about.

[IanB]

You seem to be confusing data with information. In most situations you can throw away lots of data and lose no information. That's what lossless compression is all about. In things like images you can carefully throw away a huge amount of data and lose little information. That's what lossy compression is all about.

This is not true. When you throw away lots of data you are also throwing away lots of information. Lossy compression works by selectively throwing away the least important information and keeping the most important information.

You can understand that this is true by looking to the workflow used by professionals. In image processing, the professional will always go back to the raw image capture when producing new derived images. Similarly with music, the studio engineer will always go back to the original master recordings when producing new derived works.

Lossy compression is a viable process because humans are not able to make use of all the received sensory information that comes in through the eyes and ears, so you can remove what the human physiology will filter out or miss and the subjective experience will still be good. The same is not necessarily true if a human being is not the intended consumer of the data.

[coppice]

When you throw away lots of data you are also throwing away lots of information.

Data is NOT information. If the data from a signal source looks random you can't reduce its size. Most data doesn't look random, though, and the information content of the data is quite sparse. You can compress most signals quite a lot, while losing no information at all. A lot of what makes the data from many signal sources look nearly random is actually just noise fuzzing up the data. Some mildy lossy compression schemes heavily suppress that noise, and actually clean up the signal, even though there may also be a little loss of information.

[IanB]

Data is NOT information. If the data from a signal source looks random you can't reduce its size. Most data doesn't look random, though, and the information content of the data is quite sparse. You can compress most signals quite a lot, while losing no information at all. A lot of what makes the data from many signal sources look nearly random is actually just noise fuzzing up the data. Some mildy lossy compression schemes heavily suppress that noise, and actually clean up the signal, even though there may also be a little loss of information.

First we were talking about lossy compression, and here you don't seem clear whether you are talking about lossy compression or lossless compression. Every other sentence you are hopping from one to the other. You need to be clear about what you are referring to. Lossless compression exploits redundancy in the source material. Lossy compression discards information deemed to be of low value to the consumer. They are quite different.

Keep in mind that information is in the eye of the beholder. Whether data is information or not depends on how and why it will be consumed. That "noise" that you are suppressing from the signal might actually be the real information content inserted by steganography. What you think of as the clean signal may just be the decoy to mislead you.

Never forget that all data is information. Even if the data is noise, it conveys information about the nature and character of that noise, and allows deductions about the noise sources present in the transmission path.

[rhb]

RTFM, Read The Fine Mathematics.

Really, the introductions are easy reading.  All the rest is just the fine print.  For this discussion you can skip the fine print.  If the fine print matters I'll point it out.

Shot noise, 1/f noise, popcorn noise, etc are not information unless you want to quantify them in order to characterize the performance of a device.

The distinction between what is signal and what is noise depends entirely upon the task at hand.  In this case we're trying to suppress visual noise artifacts and increase resolution.

Candes was the one who got the ball rolling with the paper on exact reconstruction.  That is not lossy compression.  It is lossless.  And the applications of sparse L1 pursuits are so broad it would take several feet of graduate level books to just give an overview.  So it is very hard to discuss sparse L1 pursuits without bouncing all over the place.

I had to read 3000 pages to get to the point I could see a picture in my head.  Sometimes that's what you have to do.  Reality doesn't fit in a 15 minute YouTube video.  The papers I linked are the easiest things to grasp.  I strongly recommend that you read the introductory prose and look at the pictures first.  That's what I would have done if I had not started out with Foucart and Rauhut which is long on equations and very short on pictures.  It was a great relief when I started reading the original papers, except perhaps for the one with the 15 page proof.  I think that made even Donoho's eyes glaze over.

Most people fail by not trying.  The thing that distinguishes PhDs and SOCOM operators is they don't quit.  Failure just means you have to start over. You only stop trying when you are dead.

[coppice]

Never forget that all data is information.

Why do you keep saying that, when the most basic principal of information theory is that it is bogus? Data is just a bunch of bits. Information is what you don't know, and can't figure out from other information you have. In the great majority of cases it is a subset, and often quite a small subset, of the data you are presented with.

[rhb]

Never forget that all data is information.

Why do you keep saying that, when the most basic principal of information theory is that it is bogus? Data is just a bunch of bits. Information is what you don't know, and can't figure out from other information you have. In the great majority of cases it is a subset, and often quite a small subset, of the data you are presented with.

A very important principle in programming is never to specify more parameters  than information.  Redundant information  leads to bugs.  Hopefully they crash the program, but all too often they lead to very subtle and difficult to diagnose errors in the results.

The seminal paper was "Communication in the Presence of Noise".  RFTM attached.

NB:  This thread is on the subject of  mathematics applicable to thermal imaging.  I started it by request following some comments I made in the HT-18 thread.  From the little recent work I've read about thermal imaging it does not appear generally to be as sophisticated as astrophotography.  In any case, sparse L1 pursuits are driving the state of the art in many disciplines.  There are many opportunities to make major advances.

[coppice]

A very important principle in programming is never to specify more parameters  than information.  Redundant information  leads to bugs.  Hopefully they crash the program, but all too often they lead to very subtle and difficult to diagnose errors in the results.

That's a good guiding rule, as long as its not followed cult like. Some values take a lot of compute to recreate each time they are needed, so calculating them once, and passing them around to all the things which need them, is a sensible efficiency move.

[rhb]

Flir has the suppression of row and column noise blanketed with patents, however, I could not find any papers characterizing the sources of row and column noise.

The most recent patent I found was granted in 2016.

I've seen mention of hacks to increase the resolution of Flir cameras.  Does anyone have any information about how Flir is handling resolution enhancements?

[Vipitis]

Flir has 3 tricks.

The main one is to use a higher resolution sensor and dumm in down in software to sell the same hardware as a cheaper model with a different model name. Meaning a whole line of Ex cameras are basically the same hardware and clever forum members found different methods of liberating the full potential by editing the firmware and configurion files. This enables only 1:1 sensor resolution, not beyond.

Next up is UltraMax, which is basically just superresolution by shaky hands implemented in their high end handheld models of the Txxx series. Effectively increasing the image resolution 4x by combining 4 images and giving you actual, real data. Some visual cameras use their sensor stabilization to do a "sensor shift" and move the sensor by 0.5 pixel pitch around when you camera is on a tripod to effectively increasing your resolution using the same method.
This is only practical on devices with a high framerate. ThermalCamera+ for FLIR One/CAT S60 has some form of superresolution built in as well.

The third trick they do degrades their images by artificial interpolating into a higher resolution for no apparent reason. This causes blurring and fake data. This is very prominent in the low resolution Lepton devices like FLIR One, when you check an image in FLIR Tools or export a .cvs for example.

For actual image enhancements based I maths, FLIR had different 'features' based on what kind of device they sell you.
MSX is their patent to overlay an visible image with edge detection(in my mind wrongly implemented as off axis optics could fix the parallax issues and a synthetic image reinterpretation using image recognition and applying gradients of temperature with bases in the thermal pixels would make this an actually useful feature) it mostly covers up the horrendes low resolution on their consumer porducts.

DDE is their digital detail enhancement patent technology found in devices for surveillance and security like military or hunting scopes/monoculars/vehicle based systems. It is a combination of standard histogram stretching and sharpening.

they also we're caught adding fake noise to low end devices to make her top enf look better. This works similar too the resolution lock down and has been bypassed by forum members.

As of any actual resolution enhancements with math tricks or computational imaging, nothing too my knowledge - but remember that this technology find military application first and the development might be much farther then publicly know. I could think of reconnaissance satellites using such advanced algorithms to get down to resolutions near Dawes limit.

[IanB]

As of any actual resolution enhancements with math tricks or computational imaging, nothing too my knowledge - but remember that this technology find military application first and the development might be much farther then publicly know. I could think of reconnaissance satellites using such advanced algorithms to get down to resolutions near Dawes limit.

Right, but the achievable resolution is always going to be limited by the physics of the imaging system. Mathematics can help you achieve the maximum potential of the system, however to go beyond that would be over-unity, and nobody should rationally believe that to be possible.

[Bud]

Flir has 3 tricks.

They also induce substantial amount of artificial noise, which the liberation process gets rid of.

[rhb]

I'm going to drop out of this thread.  Between Flir's lawyers and DoJ's lawyers  I might easily find myself with a lot of unwelcome problems. 

From the dates of the Flir patents on suppressing row and column noise it is clear that the signal processing aspect of thermal imaging is not very well developed in the private sector.  I'm certain that Raytheon and other DoD suppliers are more advanced.  But that is ITAR territory and I have no wish to go there in a public forum.

I find it frustrating to do this, but there are times when it is best not to speak of certain things.

[IanB]

I'm going to drop out of this thread.  Between Flir's lawyers and DoJ's lawyers  I might easily find myself with a lot of unwelcome problems. 

From the dates of the Flir patents on suppressing row and column noise it is clear that the signal processing aspect of thermal imaging is not very well developed in the private sector.  I'm certain that Raytheon and other DoD suppliers are more advanced.  But that is ITAR territory and I have no wish to go there in a public forum.

I find it frustrating to do this, but there are times when it is best not to speak of certain things.

With respect, you haven't gone anywhere in this thread. First we have vague, unspecific claims of miraculous mathematics. Now we have vague, unspecified fears of government persecution.

Maybe you could explain, in plain English, as if to a high school student, what is the unique and special value of "sparse L1 pursuits"? That would garner much more useful discussion than pointing people to long and technical mathematical papers without saying what it's for or how it's helpful.

Color me skeptical, but this thread has always been in danger of entering tinfoil hat territory, and you just took it there.

[helius]

IanB, with all respect, I think it is you who is derailing the thread. Why do you insist that something should be explainable in "plain English"? You remind me of the crackpots who dismiss string theory because it cannot be explained in "plain English". Or an infamous Usenet troll who refused to accept complex numbers unless they could be "explained in terms of apples."

You also err in your claims about "workflow used by professionals." It is now possible to perform edits to individual instruments within a mix using Melodyne and no longer necessary to go back to a multitrack master recording. It is also common workflow to perform photo editing at a lower resolution than either the RAW file or the final output, with plugins like ON1 Resize. These are both applications of wavelet transforms. Wavelets are fundamental tools for modern engineering and it doesn't matter that you request plain English explanations.

[eKretz]

I think the misconception is in the fact that most people consider an individual pixel as only able to record or represent a single color and luminance level. It seems that is beginning to change. Here's another relevant link:

http://news.mit.edu/2017/faster-single-pixel-camera-lensless-imaging-0330

[IanB]

I think the misconception is in the fact that most people consider an individual pixel as only able to record or represent a single color and luminance level. It seems that is beginning to change. Here's another relevant link:

http://news.mit.edu/2017/faster-single-pixel-camera-lensless-imaging-0330

Cool. Very interesting.

Also, that article does a remarkable job of explaining certain theory, and its application, and its value in simple, plain English without any complex mathematics. Kudos to the author of the article. The concept is straightforward and requires no magic. I predicted what the article was going to say before reading it.

The slight half-truth in what you wrote ( ) is that a single pixel is only able to record a single color and luminance level at a single moment in time. If you allow time to be an additional dimension you can achieve more. But, this relates back to what I said in a previous post. To bring time into the equation more effectively and to achieve advances in imaging requires (very) clever physics and engineering. It is always the physics of the imaging system that defines and constrains what you can achieve.

Mathematics, in and of itself, can only allow you to approach ever closer to the physical potential of the system (unity). Mathematics can never allow you to achieve more than that. Over-unity is not possible.

[CatalinaWOW]

While the optics is a fundamental element limiting resolution, there is some wiggle room in that limit.  Most  definitions of that limit boil down to the ability to use a simple amplitude threshold to separate two sources.

To paint a simple word picture the optics response to a point source is more or less a gaussian shaped amplitude distribution.  As two point sources are moved apart the amplitude response starts to show a dip and with enough separation that dip can easily be detected and separated from noise.  This is the conventional definition of the resolution limit of an optic.  Obviously the distribution from two point sources looks different from a single point distribution for any case other than when the two point sources are superposed.  If the point distribution of the optic is known, you can determine the separation in two dimensions from the distribution of light from two sources, even if they are closer together than the "optical resolution".

This gets more complicated when you sample the data with a detector and have a more general image than two point sources.  I haven't digested the mathematics of this L1 stuff yet, but suspect that in essence doing this more general case is what is happening.  With the benefit of computational efficiency.  Not something for nothing, just wringing all the water possible out of the towel.

[Ultrapurple]

...  I started this thread at the request of @ultrapurple who has not shown up.-

Yes, sorry I haven't been around - I have been fully committed to family activities for the last several days.

I am finding this discussion fascinating and can see how the sparse sampling techniques could make a major impact on thermal (or other intrinsically low resolution) imaging. One of my first thoughts was whether the regular array of pixels in a thermal camera could be treated as a special case of randomness and built upon from there. Similarly, using a 'randomly' coded aperture might be possible by selecting a random number of pixels from the array. (An interesting side-effect of this would be that dead pixels essentially cease to be a problem; whereas currently a sensor with 1% dead pixels is pretty much a reject, I can imagine sensors with 50% dead pixels being perfectly usable. Manufacturing yield goes way up, prices come way down).

The other exciting thing for me is that it appears that the basic principles have been shown to work, so it's 'just' a matter of turning the maths into applied software and cobbling together suitable hardware to run it on. So what if it takes a rack full of today's CPUs to run in real time? It won't be long before someone obeys Moore's Law and shrinks it into a single package. (At the start of what I laughably refer to as my 'career' I remember being really impressed with some real-time image processing avionics running on custom bit-slice processors encased in several boxes each the size and weight of two dozen bottles of wine. Nowadays the same image processing task would be considered trivial on a mid-range mobile phone - and, to my chagrin, I recently saw key parts of the system on eBay for peanuts). But I digress.

The maths of all this is well beyond me but I am keeping up with the principles well enough to see at least some of the potential. I will continue to watch and learn with much interest.

[Ultrapurple]

...Most  definitions of that limit boil down to the ability to use a simple amplitude threshold to separate two sources.

To paint a simple word picture the optics response to a point source is more or less a gaussian shaped amplitude distribution.  As two point sources are moved apart the amplitude response starts to show a dip and with enough separation that dip can easily be detected and separated from noise.  This is the conventional definition of the resolution limit of an optic.  ...

All true. However, what happens if you add a time-of-flight dimension into the imaging system? Clearly, if the two point sources are (nearly) co-located at precisely the same distance from the sensor then ToF wont help. Furthermore, in thermal imaging we're normally considering emissions from the subject itself, rather than determining how it reflects an external energy source. I suspect a 'thermal flashgun with coded aperture lighting pattern' would be tricky to engineer but then again, some clever person will probably come up with pixels that are both phase- and frequency-sensitive, or otherwise unimaginably ingenious. I await developments with awe and excitement.

Thermal imaging has come a long way in the last 20-30 years and I suspect it will develop even further.

[rhb]

If you start with Candes and Donoho in 2004 and read the English prose introductions to their papers you'll have a very good grasp of what can be done.  The proof details are important, but not particularly illuminating.

All the rest is solve Ax=y using an L1 norm.  The magic is in the properties of an L1 norm when the solution is sparse.

The method is completely general so it applies in almost any endeavor.  The constraints  are that any combination of columns of A have low correlation with any other combination of columns.  The other constraint is "sufficient" data.  Realistically, you set up and attempt to solve the problem using a library call or the equivalent.

I like using GMPL.  The equations are in a small file which is hand edited.  The data file is machine generated.  The astrophotography crowd probably already has good algorithms for image enhancement. Some of which may be essentially a sparse L1 pursuit with a different name.