Flir Ex: Realtime raw radiometric data streaming via UVC

[OrBy]

I will still try to get the camera stream same way as OrBy's so i can post processes images received using the two methods and compare the end result.

I can grab some more out of each stream and put them up for comparison later today if needed/wanted.

[Bud]

Having said that, i begin to believe that may not be difference between the E4 hacked radiometric streamed images vs screen capture streamed ones

...except perhaps swapped bytes in radiometric stream.

[tomas123]

i begin to believe that may not be difference between the E4 hacked radiometric streamed images vs screen capture streamed ones, providing i turn measurements and everything else off on the camera, and also turn MSX off so the screen only displays the thermal image.

Stream of the monitor view is nice but a raw stream is really cool (surveillance camera, measurements)
See the great video from tmbinc.
I hope that tmbinc give us the code for converting a raw stream. I see a nice application for my raspberry pi.

Don't forget that Flir Tools can grab a raw stream from E40 (switch from video to signal)

https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg377140/#msg377140

the video stream of E4 grab with the free FLIR IR Camera Player

https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg378926/#msg378926

[tomas123]

@OrBy

Your upgraded the E4 from 1.18.7 to 2.3.0.
Interesting!
Maybe the (hidden) raw stream of your E4 is activated by a switch in your config file.

Just wanted to give a quick  to janekivi and everyone else that's been working on 2.3.0.

I decided to upgrade from 1.18.7 to 2.3.0 today on my 1.0 E4.

I just flashed the 2.3.0 FIF and rebooted and it retained my "upgraded" status. Installed the 2014 menu FIF and had to add a little tweak to my e8.cfg and recrc it to get the manual adjustments working.

So far I really like the clean no measurements screen! Makes for "truer" video capture in UVC mode and allows me to "frame" my shots better! Also the thermal blending is a nice addition.

Cant wait till I have some more time to play with the advanced ed.

[OrBy]

Yes I did upgrade mine to 2.3.0

Here is my lsusb -v dump:
(I hope it's the right section required)

Code: [Select]

Bus 001 Device 002: ID 09cb:1007 
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 ?
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  idVendor           0x09cb
  idProduct          0x1007
  bcdDevice            0.00
  iManufacturer           7
  iProduct                8
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          241
    bNumInterfaces          3
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass         14 Video
      bFunctionSubClass       3 Video Interface Collection
      bFunctionProtocol       0
      iFunction              10
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      1 Video Control
      bInterfaceProtocol      0
      iInterface              2
      VideoControl Interface Descriptor:
        bLength                13
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdUVC               1.00
        wTotalLength           85
        dwClockFrequency        0.001000MHz
        bInCollection           1
        baInterfaceNr( 0)       1
      VideoControl Interface Descriptor:
        bLength                18
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             1
        wTerminalType      0x0201 Camera Sensor
        bAssocTerminal          0
        iTerminal               0
        wObjectiveFocalLengthMin      0
        wObjectiveFocalLengthMax      0
        wOcularFocalLength            0
        bControlSize                  3
        bmControls           0x00020260
          Focus (Absolute)
          Focus (Relative)
          Zoom (Absolute)
          Focus, Auto
      VideoControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             3
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bSourceID               6
        iTerminal               0
      VideoControl Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      4 (SELECTOR_UNIT)
        bUnitID                 4
        bNrInPins               1
        baSource( 0)            1
        iSelector               0
      VideoControl Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      5 (PROCESSING_UNIT)
      Warning: Descriptor too short
        bUnitID                 5
        bSourceID               4
        wMaxMultiplier          0
        bControlSize            2
        bmControls     0x00000217
          Brightness
          Contrast
          Hue
          Sharpness
          Gain
        iProcessing             0
        bmVideoStandards     0x1b
          None
          NTSC - 525/60
          SECAM - 625/50
          NTSC - 625/50
      VideoControl Interface Descriptor:
        bLength                27
        bDescriptorType        36
        bDescriptorSubtype      6 (EXTENSION_UNIT)
        bUnitID                 6
        guidExtensionCode         {d41f59fa-5094-463a-b3bb-e7858a831fa3}
        bNumControl             4
        bNrPins                 1
        baSourceID( 0)          5
        bControlSize            2
        bmControls( 0)       0x0f
        bmControls( 1)       0x00
        iExtension              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              16
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      2 Video Streaming
      bInterfaceProtocol      0
      iInterface              0
      VideoStreaming Interface Descriptor:
        bLength                            14
        bDescriptorType                    36
        bDescriptorSubtype                  1 (INPUT_HEADER)
        bNumFormats                         1
        wTotalLength                       79
        bEndPointAddress                  130
        bmInfo                              0
        bTerminalLink                       3
        bStillCaptureMethod                 0
        bTriggerSupport                     0
        bTriggerUsage                       0
        bControlSize                        1
        bmaControls( 0)                    27
      VideoStreaming Interface Descriptor:
        bLength                            27
        bDescriptorType                    36
        bDescriptorSubtype                  4 (FORMAT_UNCOMPRESSED)
        bFormatIndex                        1
        bNumFrameDescriptors                1
        guidFormat                            {59555932-0000-1000-8000-00aa00389b71}
        bBitsPerPixel                      16
        bDefaultFrameIndex                  1
        bAspectRatioX                       0
        bAspectRatioY                       0
        bmInterlaceFlags                 0x00
          Interlaced stream or variable: No
          Fields per frame: 2 fields
          Field 1 first: No
          Field pattern: Field 1 only
          bCopyProtect                      0
      VideoStreaming Interface Descriptor:
        bLength                            38
        bDescriptorType                    36
        bDescriptorSubtype                  5 (FRAME_UNCOMPRESSED)
        bFrameIndex                         1
        bmCapabilities                   0x03
          Still image supported
          Fixed frame-rate
        wWidth                            320
        wHeight                           240
        dwMinBitRate                   912384
        dwMaxBitRate                   912384
        dwMaxVideoFrameBufferSize      153600
        dwDefaultFrameInterval        2666664
        bFrameIntervalType                  3
        dwFrameInterval( 0)            666666
        dwFrameInterval( 1)           1333332
        dwFrameInterval( 2)           2666664
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0

And here are the contents of my e8.cfg:

Code: [Select]

.caps entry
.caps.config entry
.caps.config.name text "app E8"
.caps.config.revision text "1.0"
.caps.config.image entry
.caps.config.image.framegrab entry
.caps.config.image.framegrab.fusion entry
.caps.config.image.framegrab.fusion.enabled bool true
.caps.config.image.framegrab.fusion.pip entry
.caps.config.image.framegrab.fusion.pip.enabled bool true
.caps.config.image.framegrab.fusion.hcf entry
.caps.config.image.framegrab.fusion.hcf.enabled bool true
.caps.config.image.services entry
.caps.config.image.services.store entry
.caps.config.image.services.store.enabled bool true
.caps.config.image.services.store.radiometric entry
.caps.config.image.services.store.radiometric.enabled bool true
.caps.config.image.services.store.incompatible entry
.caps.config.image.services.store.incompatible.enabled bool false
.caps.config.image.services.store.incompatible.level int32 0
.caps.config.image.settings entry
.caps.config.image.settings.enabled bool true
.caps.config.image.settings.IRwidth int32 320
.caps.config.image.settings.IRheight int32 240
.caps.config.image.sysimg entry
.caps.config.image.sysimg.alarms entry
.caps.config.image.sysimg.alarms.enabled bool true
.caps.config.image.sysimg.alarms.measfunc entry
.caps.config.image.sysimg.alarms.measfunc.enabled bool true
.caps.config.image.sysimg.alarms.measfunc.maxCount int32 3
.caps.config.image.sysimg.alarms.humidity entry
.caps.config.image.sysimg.alarms.humidity.enabled bool true
.caps.config.image.sysimg.alarms.humidity.maxCount int32 1
.caps.config.image.sysimg.alarms.insulation entry
.caps.config.image.sysimg.alarms.insulation.enabled bool true
.caps.config.image.sysimg.alarms.insulation.maxCount int32 1
.caps.config.image.sysimg.irMarkers entry
.caps.config.image.sysimg.irMarkers.enabled bool true
.caps.config.image.sysimg.irMarkers.spot entry
.caps.config.image.sysimg.irMarkers.spot.enabled bool false
.caps.config.image.sysimg.irMarkers.spot.maxCount int32 0
.caps.config.image.sysimg.irMarkers.arrow entry
.caps.config.image.sysimg.irMarkers.arrow.enabled bool true
.caps.config.image.sysimg.irMarkers.arrow.maxCount int32 4
.caps.config.image.sysimg.irMarkers.box entry
.caps.config.image.sysimg.irMarkers.box.enabled bool false
.caps.config.image.sysimg.irMarkers.box.maxCount int32 0
.caps.config.image.sysimg.measureFuncs entry
.caps.config.image.sysimg.measureFuncs.enabled bool true
.caps.config.image.sysimg.measureFuncs.diff entry
.caps.config.image.sysimg.measureFuncs.diff.enabled bool true
.caps.config.image.sysimg.measureFuncs.diff.maxCount int32 1
.caps.config.image.sysimg.measureFuncs.diff.calcMask int32 65526
.caps.config.image.sysimg.measureFuncs.isotherm entry
.caps.config.image.sysimg.measureFuncs.isotherm.enabled bool true
.caps.config.image.sysimg.measureFuncs.isotherm.calcMask int32 20
.caps.config.image.sysimg.measureFuncs.isotherm.dual bool false
.caps.config.image.sysimg.measureFuncs.isotherm.fixScale bool false
.caps.config.image.sysimg.measureFuncs.isotherm.interval bool true
.caps.config.image.sysimg.measureFuncs.isotherm.invInterval bool false
.caps.config.image.sysimg.measureFuncs.isotherm.maxCount int32 1
.caps.config.image.sysimg.measureFuncs.mbox entry
.caps.config.image.sysimg.measureFuncs.mbox.enabled bool true
.caps.config.image.sysimg.measureFuncs.mbox.calcMask int32 1924
.caps.config.image.sysimg.measureFuncs.mbox.maxCount int32 5
.caps.config.image.sysimg.measureFuncs.mcircle entry
.caps.config.image.sysimg.measureFuncs.mcircle.enabled bool false
.caps.config.image.sysimg.measureFuncs.mcircle.calcMask int32 1924
.caps.config.image.sysimg.measureFuncs.mcircle.maxCount int32 0
.caps.config.image.sysimg.measureFuncs.mline entry
.caps.config.image.sysimg.measureFuncs.mline.enabled bool false
.caps.config.image.sysimg.measureFuncs.mline.calcMask int32 1924
.caps.config.image.sysimg.measureFuncs.mline.maxCount int32 0
.caps.config.image.sysimg.measureFuncs.reftemp entry
.caps.config.image.sysimg.measureFuncs.reftemp.enabled bool true
.caps.config.image.sysimg.measureFuncs.reftemp.calcMask int32 1924
.caps.config.image.sysimg.measureFuncs.reftemp.maxCount int32 1
.caps.config.image.sysimg.measureFuncs.script entry
.caps.config.image.sysimg.measureFuncs.script.enabled false
.caps.config.image.sysimg.measureFuncs.script.maxCount int32 0
.caps.config.image.sysimg.measureFuncs.spot entry
.caps.config.image.sysimg.measureFuncs.spot.enabled bool true
.caps.config.image.sysimg.measureFuncs.spot.calcMask int32 514
.caps.config.image.sysimg.measureFuncs.spot.maxCount int32 5
.caps.config.image.sysimg.visualMarkers entry
.caps.config.image.sysimg.visualMarkers.enabled bool true
.caps.config.image.sysimg.visualMarkers.spot entry
.caps.config.image.sysimg.visualMarkers.spot.enabled bool false
.caps.config.image.sysimg.visualMarkers.spot.maxCount int32 0
.caps.config.image.sysimg.visualMarkers.arrow entry
.caps.config.image.sysimg.visualMarkers.arrow.enabled bool true
.caps.config.image.sysimg.visualMarkers.arrow.maxCount int32 4
.caps.config.image.sysimg.visualMarkers.box entry
.caps.config.image.sysimg.visualMarkers.box.enabled bool false
.caps.config.image.sysimg.visualMarkers.box.maxCount int32 0
.caps.config.image.contadj entry
.caps.config.image.contadj.minSpanFactor entry
.caps.config.image.contadj.minSpanFactor.enabled bool true
.caps.config.image.contadj.minSpanFactor.factorAuto double 2.0
.caps.config.image.contadj.minSpanFactor.factorManual double 2.0
.caps.config.image.targetNoise entry
.caps.config.image.targetNoise.enabled bool false
.caps.config.image.targetNoise.targetNoiseMk int32 0
.caps.config.image.zoom entry
.caps.config.image.zoom.enabled bool true
.caps.config.image.zoom.maxFactor double 8
.caps.config.system entry
.caps.config.system.focus entry
.caps.config.system.focus.laser entry
.caps.config.system.focus.laser.updateFocus entry
.caps.config.system.focus.laser.updateFocus.enabled bool true
.caps.config.ui.image entry
.caps.config.ui.image.adjust entry
.caps.config.ui.image.adjust.enabled bool true
.caps.config.ui.image.adjust.manual bool true
.caps.config.ui entry
.caps.config.ui.fusion entry
.caps.config.ui.fusion.PIP entry
.caps.config.ui.fusion.PIP.enabled bool true
.caps.hw entry
.caps.hw.sdcard entry
.caps.hw.sdcard.enabled bool false


[Bud]

I followed tomas123's steps to re-compile libuvc library and now it works for me too (thanks tomas123 for posting the compilation log, I think I can learn Chinese faster than Linux ).

Attached is a frame from the video stream, loaded in FLIR Tools and some measurements enabled. All makes sense and matches what is measured by the camera. For the record, the camera is a native E4 fw 2.3.0 doctored for resolution and enhanced menu.

So yes the patch works, it is  possible to stream radiometric video from E4 (oh well, for now just in a file and using Linux), thanks user tmbinc 

[tomas123]

Great!
Please describe all steps, how you can load a frame in Flir Tools and attach a splitted single raw frame (as .zip)

I think, I must make a firmware update from 1.19 to 2.3.0
don't touch a running system 

[Bud]

All as previously described:

Pre-requisits for image processing software:

- imagemagick
- php
- splitjpg.php script
    source: https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg348715/#msg348715
- a sample image from the same E4 camera
- FLIR Tools

workflow:

- capture a RAW video
- split to individual frames

Code: [Select]

split -b 153600 test1.raw
- convert the required frame (in this example the first frame) to a RAW png

Code: [Select]

convert -depth 16 -size 320x240  gray:xaa raw1.png
convert the RAW png to radiometric jpg (for a template use a sample jpg image, e.g. FLIR0123.jpg, from the same camera:

Code: [Select]

php splitjpg.php -i FLIR0123.jpg -r raw1.png -o test1.jpgnote: after this step the resulting test1.jpg preview will appear the same as the template image, do not worry: the actual RAW data is hidden behind the image.

Now load this JPG into FLIR Tools to rebuild the preview image, place temperature measurements, apply a desired palette and resave.

I do video capture, split and convert on a Ubuntu VM, everything else on a Windows 7.

[Bud]

Found a great little utility called ImageJ

http://imagej.nih.gov/ij/

Holy sh!t, have not seen it before, it is a jewel. For the purpose of this thread it can directly open raw files created by tmbinc's software and resave in a bunch of different formats including individual split frames. Also has a pile of built-in pallettes (Look Up Table they call it) and can colorize the video and even save the video in AVI format ready to be played ! 

A sample is attached - a radiometric stream capture from the E4, colorized using ImageJ.

test13.avi  <- click here

With this sort of post processing software radiometric streaming from E4 all of a sudden makes sense.

[tomas123]

  Great!!!
ImageJ read my E40 raw stream with little endian byte order and 3840 Byte Flir-FFF-Header between frames

@Bud
I linked your last two posts in my E4 link list:
https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg342072/#msg342072
see topic 15

Mike linked my post to the the head of the infinite thread
https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/
"Tomas123's links on post- processing and measurements using E4 images"


Please add in your second last post a link to my splitjpg.php source
https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg348715/#msg348715

@tmbinc
I hope you can give us your Fuchsjagd code snippets 

[Bud]

@Bud
Please add in your second last post a link to my splitjpg.php source
https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg348715/#msg348715

Done.

I wish ImageJ could combine multiple frames in a single image for superresolution, but have not figured out yet if it can.
A newer version is here:

http://fiji.sc/Downloads

Better palettes, +tons of image manipulation menus, a holy grail of image processing 

[tomas123]

a little bit off topic: superresolution with RAW thermal video

I think, you know my posts to superresolution.
(see my link list #17 -> https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg363688/#msg363688)

I have had good experiences with:

- Registax (sharp with wavelets)

- AviStack  (sharp with wavelets)

- Photoshop CS3 -> automate -> merge to HDR (automat. align images)
  -> quality like Registax/AviStax (CS3 works better as CS5)

... do not expect too much and resize your images/video before stacking

[Bud]

I wish ImageJ could combine multiple frames in a single image for superresolution, but have not figured out yet if it can.
A newer version is here:

http://fiji.sc/Downloads

Better palettes, +tons of image manipulation menus, a holy grail of image processing 

OK I think there is a few ways to experiment with superresolution in ImageJ. For the purpose of this topic the workflow to create a combined image from a sequence of video frames may look as follows:

- capture a few seconds of a raw video as already described above
- open the video in ImageJ (File->Import->RAW)
- if necessary, delete bad or unwanted frames (use scroll bar to select a frame, then "Image->Stacks->Delete Slice"
- combine the frames (Image->Stacks->Z Project)

[tmbinc]

Sorry for not keeping up with this thread.

The idea is that even though the E4 doesn't announce the bFormatIndex=2 RAW stream via the USB descriptor, it still allows to select it. The code that restricts the raw mode based on the camera type does cripple the descriptor, but it doesn't prevent you from still selecting it.

The libuvc patch is what I've posted (i just double-checked against what's still on my raspi2), I applied it on top of https://github.com/ktossell/libuvc/commits/6ff39f17e3f3a26f437bc6b3c09cc6c289db7891 . It does this in a very lame and stupid way by switching it back and forth, to make the library take the original stream descriptor, but send the right UVC command.

I attached the "fuchsjags" source, but it's really not great - it calculates min and max, scales that to 0..255, inserts the texts and the pointer, converts it to ARGB for the framebuffer, and also dumps the raw and converted values into files. I have a different script that used avconv to encode it to MP4 ("avconv -f rawvideo -pix_fmt gray -s:v 320x240 -r 9 -i /mnt/fox/r1_proc.raw -c:v libx264 /mnt/fox/output.mp4").

[tomas123]

Thank You 

My next project name is Marderjagd 

[tomas123]

I think, I must make a firmware update from 1.19.8 to 2.3.0
don't touch a running system 

successful upgraded the E4 from 1.19.8 to 2.3.0 with Flir Tools
The resolution hack survived the update (nothing else to do).

the RAW stream grabbing now works fine

[Bud]

Great, so RAW streaming did not work in 1.19.8 ...

[Bud]

@tomas123

I wonder if you then could take a challenge and compile the other sample from libuvc which supposedly creates a window and displays the streamed video (source modified for 320x240x15 is attached). It requires a library called opencv to be installed. I tried but as I said my Chinese is much better than my Linux, no matter how hard I tried I got the dreaded opencv "Rebuild the library with Windows, GTK+ 2.x or Carbon support" error. I am sure I am missing something obvious but I just do not know what.

If this code works and if combined with tmbinc's code this may become a usable program for E4 that displays, saves and processes E4 radiometric video.

[tmbinc]

By the way, my test tool already shows the output on a framebuffer. It's not great and was just meant for debugging... Should be easy though to add a LUT and some color scheme, as well as "calibrated" temperature. I'm not sure though that the radiometric metadata is available.

I also tried doing a 2D-FFT to assist in fine-tuning focus. But the results haven't been that useful. Does someone have a better idea how to do this? (I.e. how to evaluate focus based on image processing)

[Zucca]

l upgraded the E4 from 1.19.8 to 2.3.0 with Flir Tools
The resolution hack survived the update (nothing else to do).

Temped to do the same a long time ago, didn´t have time yet:

https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg607838/#msg607838

and do the janekivi menu mod. Hope the raw capturing will work regardless the menu mod.

[Bud]

I also tried doing a 2D-FFT to assist in fine-tuning focus. But the results haven't been that useful. Does someone have a better idea how to do this? (I.e. how to evaluate focus based on image processing)

The opencv library is a tool for computer vision, so I'd think it may have something in it for what you need.

[tomas123]

@tomas123

I wonder if you then could take a challenge and compile the other sample from libuvc which supposedly creates a window and displays the streamed video (source modified for 320x240x15 is attached). It requires a library called opencv to be installed. I tried but as I said my Chinese is much better than my Linux, no matter how hard I tried I got the dreaded opencv "Rebuild the library with Windows, GTK+ 2.x or Carbon support" error. I am sure I am missing something obvious but I just do not know what.

If this code works and if combined with tmbinc's code this may become a usable program for E4 that displays, saves and processes E4 radiometric video.

Hi Bud,

your code snippet works fine, no changes are necessary 

edit 30.08.15: found the source https://github.com/ktossell/libuvc/blob/master/src/test.c   

the steps:

(1) check version

Code: [Select]

$ lsb_release -a
Description: Ubuntu 14.04.1 LTS

(2) compile opencv
use this tutorial
http://www.bogotobogo.com/OpenCV/opencv_3_tutorial_ubuntu14_install_cmake.php

Code: [Select]

$ mkdir OpenCV
$ cd OpenCV
$ git clone [url]https://github.com/Itseez/opencv.git[/url]
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON ..
$ make
$ sudo make install
$ sudo ldconfig

(3) compile your sample
// ignore warnings
// note http://stackoverflow.com/questions/9614872/opencv-application-not-library-linking-errors

Code: [Select]

$ /usr/bin/cc  -o mytest mytest.c -luvc `pkg-config --cflags --libs opencv`
mytest.c: In function ‘cb’:
mytest.c:50:66: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
   printf("callback! length = %u, ptr = %d\n", frame->data_bytes, (int) ptr);
                                                                  ^
mytest.c:50:3: warning: format ‘%u’ expects argument of type ‘unsigned int’, but argument 2 has type ‘size_t’ [-Wformat=]
   printf("callback! length = %u, ptr = %d\n", frame->data_bytes, (int) ptr);

(4) run  your code

Code: [Select]

$ sudo ./mytest
UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x240
  bit rate: 912384-912384
  max frame size: 153600
  default interval: 1/3
  interval[0]: 1/15
  interval[1]: 1/7
  interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming for 10 seconds...
set_ae_mode: Pipe (-9)
callback! length = 153600, ptr = -886544208
set_exp_abs: Pipe (-9)
init done
opengl support available
callback! length = 157690, ptr = -886544208
callback! length = 157690, ptr = -886544208
callback! length = 157690, ptr = -886544208
callback! length = 157690, ptr = -886544208

and you get a window with a nice false color video 

[Ben321]

[Apologize if this is common knowledge and I just missed the memo. As everyone knows, the main thread is _huge_ and there was some talk about this, but I didn't find the final verdict of whether it's possible or not.]

I was trying to catch a fox that ate our pet rabbits from our garden on camera (people say they return to the scene of the crime). I wasn't successful (yet!), but I learned to hate the FLIR UVC implementation since it's randomly stops working during the night.

In an attempt to rectify the situation, I looked at the UVC implementation on the camera, and figured out that there's a dormant function to stream 9Hz raw radiometric data instead of just capturing the screen. With Windows and the regular UVC drivers, the best I could capture was the screen with 3.3Hz (for whatever reason; 15Hz should be possible) at (effectively) 8-bit over a pre-selected temp range.

The catch is that the UVC driver on the Flir explicitely disables radiometric streaming for the Z3 (=Flir Ex). It enables it for other cameras, though - i think the Flir Ex0 suports it out-of-the-box. The UVC stack is implemented in usbfnvideo.dll, which unfortunately is part of nk.bin, so it can't be easily patched (at least, I can't with my limited WinCE knowledge and my unwillingness to brick the E4).

The great thing is that FLIR sucks at implementing device restrictions; they have a dedicated grabber thread for the Flir Ex-series, and it _does_ indeed implement grabbing the raw data (via FVD1: instead of capturing the screen), and they _let_ you select capturing that (via setting bFormatIndex in the stream control), but they don't announce the ability for that in the UVC descriptors, so any "well-behaving" UVC client will not allow you to select it.

I've hacked libuvc (which is based on libusb) and - tada, it just worked. I patched libuvc pretty crudely so I'm sure there's a better way to do it. I've attached the patch and the crude sample (which just dumps out raw data into a file).

Let me know if you find this useful.

Cool, but could you please attach the compiled file instead of the C source code and patch? I want a binary to use directly. I don't have much experience with C programming, but I really would like to get to use this (I'm considering getting a Flir E4, part of the Flir Ex series, but whether or not I get it will depend on what features I can access, via EASY to use hacks, and having to compile a hack is NOT easy). So can you pleas compile this, and then post the compiled binary file here?

[tomas123]

this a generic linux source code and you must compile it for your CPU (ARM / Intel etc) and your Linux version
tmbinc compiled the code on a raspberry pi 2 (quad-core ARM7) 

you need a firmware 2.3.0

read my step by step tutorial for ubuntu x64
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg729604/#msg729604[/url

[tomas123]

@Bud

just for fun I changed the sample code from libuvc for 16 Bit RAW and appended the auto-level code from tmbinc
https://github.com/ktossell/libuvc/blob/master/src/test.c

compile the code like above described

Code: [Select]

# /usr/bin/cc  -o mytest mytest.c -luvc `pkg-config --cflags --libs opencv`
... some warnings

# sudo ./mytest
UVC initialized
Device found
Device opened
...

the source:

Code: [Select]

/*********************************************************************
* Software License Agreement (BSD License)
*
*  Copyright (C) 2010-2012 Ken Tossell
*  All rights reserved.
*
*  Redistribution and use in source and binary forms, with or without
*  modification, are permitted provided that the following conditions
*  are met:
*
*   * Redistributions of source code must retain the above copyright
*     notice, this list of conditions and the following disclaimer.
*   * Redistributions in binary form must reproduce the above
*     copyright notice, this list of conditions and the following
*     disclaimer in the documentation and/or other materials provided
*     with the distribution.
*   * Neither the name of the author nor other contributors may be
*     used to endorse or promote products derived from this software
*     without specific prior written permission.
*
*  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
*  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
*  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
*  FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
*  COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
*  INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
*  BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
*  LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
*  CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
*  LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
*  ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
*  POSSIBILITY OF SUCH DAMAGE.
*********************************************************************/
#include <stdio.h>
#include <opencv/highgui.h>

#include "libuvc/libuvc.h"

struct cb_context {
  FILE *out;
  struct timeval tv_start;
  int frames;
};

void cb(uvc_frame_t *frame, void *ptr) {
  uvc_frame_t *rgb;
  uvc_error_t ret;
  IplImage* cvImg;

  printf("callback! length = %u, ptr = %d\n", frame->data_bytes, (int) ptr);

  rgb = uvc_allocate_frame(frame->width * frame->height * 3);
  if (!rgb) {
    printf("unable to allocate rgb frame!");
    return;
  }
  ret = uvc_any2rgb(frame, rgb);
  if (ret) {
    uvc_perror(ret, "uvc_any2rgb");
    uvc_free_frame(rgb);
    return;
  }

  cvImg = cvCreateImageHeader(
      cvSize(rgb->width, rgb->height),
      IPL_DEPTH_8U,
      3);

  unsigned short *pix = frame->data;
  unsigned char *fb_proc;
  fb_proc = malloc(320 * 240);
  int i, x, y;
 
   // calc min, max, rms
  int min = 0x10000, max = 0;
  for (i = 0; i < 320 * 240; ++i) {
    if (pix[i] < min) min = pix[i];
    if (pix[i] > max) { max = pix[i]; }
  }
  int scale = 0x10000 / (max - min);

  // stretch level and shift 16 to 8 Bit 
  for (y = 0; y < 240; ++y)
  {
    for (x = 0; x < 320; ++x) {
      int v = (pix[y * 320 + x] - min) * scale >> 8;
      fb_proc[y * 320 + x] = v;
    }
  }
 
 
  CvSize image_size;
  image_size.height = 240;
  image_size.width = 320;
  int channels = 1;
  IplImage *image = cvCreateImageHeader(image_size, IPL_DEPTH_8U, channels);
  cvSetData(image, fb_proc, image->widthStep);
 
  cvNamedWindow("Test", CV_WINDOW_AUTOSIZE);
  cvShowImage("Test", image);
 
  cvWaitKey(10);

  cvReleaseImageHeader(&cvImg);

  uvc_free_frame(rgb);
}

int main(int argc, char **argv) {
  uvc_context_t *ctx;
  uvc_device_t *dev;
  uvc_device_handle_t *devh;
  uvc_stream_ctrl_t ctrl;
  uvc_error_t res;
  struct cb_context cb_ctx = {0};
  /* Initialize a UVC service context. Libuvc will set up its own libusb
   * context. Replace NULL with a libusb_context pointer to run libuvc
   * from an existing libusb context. */
  res = uvc_init(&ctx, NULL);

  if (res < 0) {
    uvc_perror(res, "uvc_init");
    return res;
  }

  puts("UVC initialized");

  res = uvc_find_device(
      ctx, &dev,
      0, 0, NULL);

  if (res < 0) {
    uvc_perror(res, "uvc_find_device");
  } else {
    puts("Device found");

    res = uvc_open(dev, &devh);

    if (res < 0) {
      uvc_perror(res, "uvc_open");
    } else {
      puts("Device opened");

      uvc_print_diag(devh, stderr);
    /* Try to negotiate a 640x480 30 fps YUYV stream profile */
      res = uvc_get_stream_ctrl_format_size(
          devh, &ctrl, UVC_FRAME_FORMAT_YUYV, 320, 240, 15  /* YUV 422, aka YUV 4:2:2. try _COMPRESSED */
      );
  /* Print out the result */
      uvc_print_stream_ctrl(&ctrl, stderr);

      if (res < 0) {
        uvc_perror(res, "get_mode");/* device doesn't provide a matching stream */
      } else {
       /* Start the video stream. The library will call user function cb:
         *   cb(frame, (void*) 12345)
         */
        res = uvc_start_streaming(devh, &ctrl, cb, &cb_ctx, 0);

        if (res < 0) {
          uvc_perror(res, "start_streaming"); /* unable to start stream */
        } else {
          puts("Streaming for 10 seconds...");
          uvc_error_t resAEMODE = uvc_set_ae_mode(devh, 1);
          uvc_perror(resAEMODE, "set_ae_mode");
          int i;
          for (i = 1; i <= 10; i++) {
            /* uvc_error_t resPT = uvc_set_pantilt_abs(devh, i * 20 * 3600, 0); */
            /* uvc_perror(resPT, "set_pt_abs"); */
            uvc_error_t resEXP = uvc_set_exposure_abs(devh, 20 + i * 5);
            uvc_perror(resEXP, "set_exp_abs");
           
            sleep(1);
          }
          sleep(20);
          uvc_stop_streaming(devh);
  puts("Done streaming.");
        }
      }

      uvc_close(devh);
      puts("Device closed");
    }

    uvc_unref_device(dev);
  }

  uvc_exit(ctx);
  puts("UVC exited");

  return 0;
}


RAW live stream:



@ tmbinc

What do you mean with this code snippset from fuchsjagd?
It's like a 20x20 pixel watermark.
You can see it in your video at the bottom on the left side.

Code: [Select]

  unsigned short *pix = frame->data;
  int x, y;
 
  for (y = 160; y < 180; ++y) {
    for (x = 60; x < 80; ++x) {
      pix[y * 320 + x] = pix[y * 320 + 60];
    }
  }
Thank you for the fuchsjagd code
I changed it for Ubuntu x64, only the frame buffer part it's a little bit tricky.