Flir Ex: Realtime raw radiometric data streaming via UVC

[tmbinc]

[Apologize if this is common knowledge and I just missed the memo. As everyone knows, the main thread is _huge_ and there was some talk about this, but I didn't find the final verdict of whether it's possible or not.]

I was trying to catch a fox that ate our pet rabbits from our garden on camera (people say they return to the scene of the crime). I wasn't successful (yet!), but I learned to hate the FLIR UVC implementation since it's randomly stops working during the night.

In an attempt to rectify the situation, I looked at the UVC implementation on the camera, and figured out that there's a dormant function to stream 9Hz raw radiometric data instead of just capturing the screen. With Windows and the regular UVC drivers, the best I could capture was the screen with 3.3Hz (for whatever reason; 15Hz should be possible) at (effectively) 8-bit over a pre-selected temp range.

The catch is that the UVC driver on the Flir explicitely disables radiometric streaming for the Z3 (=Flir Ex). It enables it for other cameras, though - i think the Flir Ex0 suports it out-of-the-box. The UVC stack is implemented in usbfnvideo.dll, which unfortunately is part of nk.bin, so it can't be easily patched (at least, I can't with my limited WinCE knowledge and my unwillingness to brick the E4).

The great thing is that FLIR sucks at implementing device restrictions; they have a dedicated grabber thread for the Flir Ex-series, and it _does_ indeed implement grabbing the raw data (via FVD1: instead of capturing the screen), and they _let_ you select capturing that (via setting bFormatIndex in the stream control), but they don't announce the ability for that in the UVC descriptors, so any "well-behaving" UVC client will not allow you to select it.

I've hacked libuvc (which is based on libusb) and - tada, it just worked. I patched libuvc pretty crudely so I'm sure there's a better way to do it. I've attached the patch and the crude sample (which just dumps out raw data into a file).

Let me know if you find this useful.

[OrBy]

First time I have seen any info on this myself - nice work!
Would love to see what the output can turn out like.

[tmbinc]

A few hundred lines of code later (simply record all frames with a max temp of higher than some threshold, re-mapping to 8-bit, then encoding as mp4), I got a good picture of the beast:



Mission accomplished.

[Bud]

Hi tmbinc, if you could post a compiled executable and a patched dll that would help us try. 

[tomas123]

edit 08.09.2015
----------------------link list for this thread: ----------------------

You need a E4 firmware 2.3.0!!  FW 1.19.8  doesn't work (see post below)

source code for above video fuchsjagd
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg736344/#msg736344

calculate temperature from 16 bit RAW values
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg748218/#msg748218
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg744673/#msg744673

rebuild a radiometric flir jpeg from a single RAW frame
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg732177/#msg732177

use ImageJ for reading a RAW video file
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg732349/#msg732349

stream RAW thermal 16-bit video and display a rescaled 8-bit video in a OpenCV window:
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg749464/#msg749464
source code:  https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg748654/#msg748654


stream a  8-bit video to V4L2 loopback devices ( /dev/video0 ):
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg756530/#msg756530

an older post for streaming the E4 screen video to a raspberry pi (UVC, no RAW video)
https://www.eevblog.com/forum/testgear/capturing-video-from-flir-e4-to-portable-device/msg368083/#msg368083

---------------------- end link list  ----------------------

only for information:

I'm played around with the uvc patch, but it doesn't work with my E4 (RAW Video).

E4
Modell: E4 1.1
Software: 1.19.8 

first compile patched UVC library

Code: [Select]

$ uname -a
Linux ubu64 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ apt-get install CMake

$ apt --installed list| grep -i usb
libusb-0.1-4/trusty,now 2:0.1.12-23.3ubuntu1 amd64 [installed]
libusb-1.0-0/trusty,now 2:1.0.17-1ubuntu2 amd64 [installed]

$ sudo apt-get update && sudo apt-get install build-essential
$ sudo apt-get install  libjpeg-dev

$ sudo apt-get install libusb-1.0-0-dev

$ git clone [url]https://github.com/ktossell/libuvc[/url]

// don't forget to apply the patch from tmbinc in /src/
$ patch < libuvc.patch
patching file stream.c

$ cd libuvc
$ mkdir build
$ cd build
$ cmake ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- No build type selected, default to Release
-- No target type selected, default to shared library
-- Building libuvc with JPEG support.
-- Configuring done
-- Generating done
-- Build files have been written to: ../flir/libuvc/build


$ make
Scanning dependencies of target uvc
[ 11%] Building C object CMakeFiles/uvc.dir/src/ctrl.c.o
[ 22%] Building C object CMakeFiles/uvc.dir/src/ctrl-gen.c.o
[ 33%] Building C object CMakeFiles/uvc.dir/src/device.c.o
[ 44%] Building C object CMakeFiles/uvc.dir/src/diag.c.o
[ 55%] Building C object CMakeFiles/uvc.dir/src/frame.c.o
[ 66%] Building C object CMakeFiles/uvc.dir/src/init.c.o
[ 77%] Building C object CMakeFiles/uvc.dir/src/stream.c.o
[ 88%] Building C object CMakeFiles/uvc.dir/src/misc.c.o
[100%] Building C object CMakeFiles/uvc.dir/src/frame-mjpeg.c.o
Linking C shared library libuvc.so
[100%] Built target uvc

$ sudo make install
[100%] Built target uvc
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libuvc.so
-- Installing: /usr/local/include/libuvc/libuvc.h
-- Installing: /usr/local/include/libuvc/libuvc_config.h
-- Installing: /usr/local/lib/cmake/libuvc/libuvcConfig.cmake
-- Installing: /usr/local/lib/cmake/libuvc/libuvcConfigVersion.cmake
-- Installing: /usr/local/lib/cmake/libuvc/libuvcTargets.cmake
-- Installing: /usr/local/lib/cmake/libuvc/libuvcTargets-release.cmake

renew library cache and compile flir.c tool from user tmbinc 

Code: [Select]

$ sudo ldconfig
$ /usr/bin/cc  -o flir ../src/flir.c -luvc
check Flir E4 USB announcements

Code: [Select]

$ lsusb -v
Bus 001 Device 006: ID 09cb:1007
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 ?
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  idVendor           0x09cb
  idProduct          0x1007
  bcdDevice            0.00
  iManufacturer           7 FLIR Systems
  iProduct                8 FLIR Ex-Series
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          241
    bNumInterfaces          3
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass         14 Video
      bFunctionSubClass       3 Video Interface Collection
      bFunctionProtocol       0
      iFunction              10 FLIR USB Video
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      1 Video Control
      bInterfaceProtocol      0
      iInterface              2 FLIR Ex-Series
      VideoControl Interface Descriptor:
        bLength                13
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdUVC               1.00
        wTotalLength           85
        dwClockFrequency        0.001000MHz
        bInCollection           1
        baInterfaceNr( 0)       1
      VideoControl Interface Descriptor:
        bLength                18
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             1
        wTerminalType      0x0201 Camera Sensor
        bAssocTerminal          0
        iTerminal               0
        wObjectiveFocalLengthMin      0
        wObjectiveFocalLengthMax      0
        wOcularFocalLength            0
        bControlSize                  3
        bmControls           0x00020260
          Focus (Absolute)
          Focus (Relative)
          Zoom (Absolute)
          Focus, Auto
      VideoControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             3
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bSourceID               6
        iTerminal               0
      VideoControl Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      4 (SELECTOR_UNIT)
        bUnitID                 4
        bNrInPins               1
        baSource( 0)            1
        iSelector               0
      VideoControl Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      5 (PROCESSING_UNIT)
      Warning: Descriptor too short
        bUnitID                 5
        bSourceID               4
        wMaxMultiplier          0
        bControlSize            2
        bmControls     0x00000217
          Brightness
          Contrast
          Hue
          Sharpness
          Gain
        iProcessing             0
        bmVideoStandards     0x1b
          None
          NTSC - 525/60
          SECAM - 625/50
          NTSC - 625/50
      VideoControl Interface Descriptor:
        bLength                27
        bDescriptorType        36
        bDescriptorSubtype      6 (EXTENSION_UNIT)
        bUnitID                 6
        guidExtensionCode         {d41f59fa-5094-463a-b3bb-e7858a831fa3}
        bNumControl             4
        bNrPins                 1
        baSourceID( 0)          5
        bControlSize            2
        bmControls( 0)       0x0f
        bmControls( 1)       0x00
        iExtension              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              16
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      2 Video Streaming
      bInterfaceProtocol      0
      iInterface              0
      VideoStreaming Interface Descriptor:
        bLength                            14
        bDescriptorType                    36
        bDescriptorSubtype                  1 (INPUT_HEADER)
        bNumFormats                         1
        wTotalLength                       79
        bEndPointAddress                  130
        bmInfo                              0
        bTerminalLink                       3
        bStillCaptureMethod                 0
        bTriggerSupport                     0
        bTriggerUsage                       0
        bControlSize                        1
        bmaControls( 0)                    27
      VideoStreaming Interface Descriptor:
        bLength                            27
        bDescriptorType                    36
        bDescriptorSubtype                  4 (FORMAT_UNCOMPRESSED)
        bFormatIndex                        1
        bNumFrameDescriptors                1
        guidFormat                            {59555932-0000-1000-8000-00aa00389b71}
        bBitsPerPixel                      16
        bDefaultFrameIndex                  1
        bAspectRatioX                       0
        bAspectRatioY                       0
        bmInterlaceFlags                 0x00
          Interlaced stream or variable: No
          Fields per frame: 2 fields
          Field 1 first: No
          Field pattern: Field 1 only
          bCopyProtect                      0
      VideoStreaming Interface Descriptor:
        bLength                            38
        bDescriptorType                    36
        bDescriptorSubtype                  5 (FRAME_UNCOMPRESSED)
        bFrameIndex                         1
        bmCapabilities                   0x03
          Still image supported
          Fixed frame-rate
        wWidth                            320
        wHeight                           240
        dwMinBitRate                   912384
        dwMaxBitRate                   912384
        dwMaxVideoFrameBufferSize      153600
        dwDefaultFrameInterval        2666664
        bFrameIntervalType                  3
        dwFrameInterval( 0)            666666
        dwFrameInterval( 1)           1333332
        dwFrameInterval( 2)           2666664
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 ?
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0000
  (Bus Powered)
now grep a stream

Code: [Select]

$ sudo libuvc/flir e4-1.raw
UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
     bcdUVC: 0x0100
VideoStreaming(1):
     bEndpointAddress: 130
     Formats:
     UncompressedFormat(1)
            bits per pixel: 16
            GUID: 5955593200001000800000aa00389b71 (YUY2)
            default frame: 1
            aspect ratio: 0x0
            interlace flags: 00
            copy protect: 00
               FrameDescriptor(1)
                 capabilities: 03
                 size: 320x240
                 bit rate: 912384-912384
                 max frame size: 153600
                 default interval: 1/3
                 interval[0]: 1/15
                 interval[1]: 1/7
                 interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming...
got frame 1439397994:439760, 153600 bytes
got frame 1439397994:533467, 153600 bytes
got frame 1439397994:629556, 153600 bytes
got frame 1439397994:726339, 153600 bytes
got frame 1439397994:820088, 153600 bytes
got frame 1439397994:916235, 153600 bytes
got frame 1439397995:22432, 153600 bytes
got frame 1439397995:115699, 153600 bytes
got frame 1439397995:209889, 153600 bytes
got frame 1439397995:304196, 153600 bytes

split the stream and convert a single frame

Code: [Select]

$ split -b 153600 e4-1.raw
$ convert -size 320x240  yuv:xaa -separate 4.png
the result is the known screen image and not a RAW image:(

[tomas123]

the same process with an E40 works fine

E40
Modell E40 0.10
Software: 2.23.14

USB annoucement

Code: [Select]

$ lsusb -v

Bus 001 Device 008: ID 09cb:1005
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 ?
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  idVendor           0x09cb
  idProduct          0x1005
  bcdDevice            0.00
  iManufacturer           6 FLIR Systems
  iProduct                7 FLIR Exx Series
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          345
    bNumInterfaces          4
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xc0
      Self Powered
    MaxPower                0mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass          2 Communications
      bFunctionSubClass       2 Abstract (modem)
      bFunctionProtocol     255 Vendor Specific (MSFT RNDIS?)
      iFunction               9 FLIR Network Camera
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol    255 Vendor Specific (MSFT RNDIS?)
      iInterface              0
      CDC Header:
        bcdCDC               1.10
      CDC Call Management:
        bmCapabilities       0x00
        bDataInterface          1
      CDC ACM:
        bmCapabilities       0x00
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               4
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x03  EP 3 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         2
      bInterfaceCount         2
      bFunctionClass         14 Video
      bFunctionSubClass       3 Video Interface Collection
      bFunctionProtocol       0
      iFunction              10 FLIR USB Video
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      1 Video Control
      bInterfaceProtocol      0
      iInterface              4 FLIR Exx Series
      VideoControl Interface Descriptor:
        bLength                13
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdUVC               1.00
        wTotalLength           85
        dwClockFrequency        0.001000MHz
        bInCollection           1
        baInterfaceNr( 0)       3
      VideoControl Interface Descriptor:
        bLength                18
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             1
        wTerminalType      0x0201 Camera Sensor
        bAssocTerminal          0
        iTerminal               0
        wObjectiveFocalLengthMin      0
        wObjectiveFocalLengthMax      0
        wOcularFocalLength            0
        bControlSize                  3
        bmControls           0x00020260
          Focus (Absolute)
          Focus (Relative)
          Zoom (Absolute)
          Focus, Auto
      VideoControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             3
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bSourceID               6
        iTerminal               0
      VideoControl Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      4 (SELECTOR_UNIT)
        bUnitID                 4
        bNrInPins               1
        baSource( 0)            1
        iSelector               0
      VideoControl Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      5 (PROCESSING_UNIT)
      Warning: Descriptor too short
        bUnitID                 5
        bSourceID               4
        wMaxMultiplier          0
        bControlSize            2
        bmControls     0x00000217
          Brightness
          Contrast
          Hue
          Sharpness
          Gain
        iProcessing             0
        bmVideoStandards     0x1b
          None
          NTSC - 525/60
          SECAM - 625/50
          NTSC - 625/50
      VideoControl Interface Descriptor:
        bLength                27
        bDescriptorType        36
        bDescriptorSubtype      6 (EXTENSION_UNIT)
        bUnitID                 6
        guidExtensionCode         {d41f59fa-5094-463a-b3bb-e7858a831fa3}
        bNumControl             4
        bNrPins                 1
        baSourceID( 0)          5
        bControlSize            2
        bmControls( 0)       0x0f
        bmControls( 1)       0x00
        iExtension              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x84  EP 4 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              16
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      2 Video Streaming
      bInterfaceProtocol      0
      iInterface              0
      VideoStreaming Interface Descriptor:
        bLength                            15
        bDescriptorType                    36
        bDescriptorSubtype                  1 (INPUT_HEADER)
        bNumFormats                         2
        wTotalLength                      145
        bEndPointAddress                  133
        bmInfo                              0
        bTerminalLink                       3
        bStillCaptureMethod                 0
        bTriggerSupport                     0
        bTriggerUsage                       0
        bControlSize                        1
        bmaControls( 0)                    27
        bmaControls( 1)                    27
      VideoStreaming Interface Descriptor:
        bLength                            27
        bDescriptorType                    36
        bDescriptorSubtype                  4 (FORMAT_UNCOMPRESSED)
        bFormatIndex                        1
        bNumFrameDescriptors                1
        guidFormat                            {59555932-0000-1000-8000-00aa00389b71}
        bBitsPerPixel                      16
        bDefaultFrameIndex                  1
        bAspectRatioX                       0
        bAspectRatioY                       0
        bmInterlaceFlags                 0x00
          Interlaced stream or variable: No
          Fields per frame: 2 fields
          Field 1 first: No
          Field pattern: Field 1 only
          bCopyProtect                      0
      VideoStreaming Interface Descriptor:
        bLength                            38
        bDescriptorType                    36
        bDescriptorSubtype                  5 (FRAME_UNCOMPRESSED)
        bFrameIndex                         1
        bmCapabilities                   0x03
          Still image supported
          Fixed frame-rate
        wWidth                            320
        wHeight                           240
        dwMinBitRate                   912384
        dwMaxBitRate                   912384
        dwMaxVideoFrameBufferSize      153600
        dwDefaultFrameInterval        1333332
        bFrameIntervalType                  3
        dwFrameInterval( 0)            333333
        dwFrameInterval( 1)            666666
        dwFrameInterval( 2)           1333332
      VideoStreaming Interface Descriptor:
        bLength                            27
        bDescriptorType                    36
        bDescriptorSubtype                  4 (FORMAT_UNCOMPRESSED)
        bFormatIndex                        2
        bNumFrameDescriptors                1
        guidFormat                            {46374d30-0000-1000-8000-00aa00389b71}
        bBitsPerPixel                      16
        bDefaultFrameIndex                  1
        bAspectRatioX                       0
        bAspectRatioY                       0
        bmInterlaceFlags                 0x00
          Interlaced stream or variable: No
          Fields per frame: 2 fields
          Field 1 first: No
          Field pattern: Field 1 only
          bCopyProtect                      0
      VideoStreaming Interface Descriptor:
        bLength                            38
        bDescriptorType                    36
        bDescriptorSubtype                  5 (FRAME_UNCOMPRESSED)
        bFrameIndex                         1
        bmCapabilities                   0x03
          Still image supported
          Fixed frame-rate
        wWidth                            320
        wHeight                           246
        dwMinBitRate                   912384
        dwMaxBitRate                   912384
        dwMaxVideoFrameBufferSize      157440
        dwDefaultFrameInterval        1333332
        bFrameIntervalType                  3
        dwFrameInterval( 0)            333333
        dwFrameInterval( 1)            666666
        dwFrameInterval( 2)           1333332
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x85  EP 5 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 ?
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0001
  Self Powered
grep a stream

Code: [Select]

$ sudo libuvc/flir e40.raw
UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1004/0a7f4000-c632-0000-0000-000000000000) ---
Status: idle
VideoControl:
     bcdUVC: 0x0100
VideoStreaming(1):
     bEndpointAddress: 130
     Formats:
     UncompressedFormat(1)
            bits per pixel: 16
            GUID: 5955593200001000800000aa00389b71 (YUY2)
            default frame: 1
            aspect ratio: 0x0
            interlace flags: 00
            copy protect: 00
               FrameDescriptor(1)
                 capabilities: 03
                 size: 320x240
                 bit rate: 912384-912384
                 max frame size: 153600
                 default interval: 1/7
                 interval[0]: 1/30
                 interval[1]: 1/15
                 interval[2]: 1/7
     UncompressedFormat(2)
            bits per pixel: 16
            GUID: 46374d3000001000800000aa00389b71 (F7M0)
            default frame: 1
            aspect ratio: 0x0
            interlace flags: 00
            copy protect: 00
               FrameDescriptor(1)
                 capabilities: 03
                 size: 320x246
                 bit rate: 912384-912384
                 max frame size: 157440
                 default interval: 1/7
                 interval[0]: 1/30
                 interval[1]: 1/15
                 interval[2]: 1/7
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 157440
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming...
got frame 1439394751:346415, 157440 bytes
got frame 1439394751:411283, 157440 bytes
got frame 1439394751:479331, 157440 bytes
got frame 1439394751:546262, 157440 bytes
got frame 1439394751:610415, 157440 bytes
got frame 1439394751:679598, 157440 bytes
got frame 1439394751:741424, 157440 bytes
got frame 1439394751:810494, 157440 bytes
got frame 1439394751:874681, 157440 bytes
got frame 1439394751:938867, 157440 bytes
....
the streams contains RAW FFF frames

Code: [Select]

$ hexdump -C  e40.raw
00000000  f4 48 f6 48 f7 48 03 49  04 49 02 49 01 49 00 49  |.H.H.H.I.I.I.I.I|
00000010  14 49 fc 48 ff 48 04 49  fe 48 0a 49 fd 48 0d 49  |.I.H.H.I.H.I.H.I|

...
000257f0  09 49 19 49 18 49 0e 49  0e 49 0c 49 13 49 32 49  |.I.I.I.I.I.I.I2I|
00025800  46 46 46 00 55 56 43 00  00 00 00 00 00 00 00 00  |FFF.UVC.........|
00025810  00 00 00 00 64 00 00 00  40 00 00 00 02 00 00 00  |....d...@.......|
split the stream and convert a frame

Code: [Select]

$ split -b 157440 e40.raw
$ convert -depth 16 -size 320x240 -auto-level gray:xaa 8.png
it's a RAW image 


check frame rate with exiftool (15 fps)

Code: [Select]

$ dd if=e40.raw of=e40allframe.ff bs=153600 skip=1

$ exiftool -Date* x*
======== xaa
Date/Time Original              : 2015:08:12 17:39:48.413+01:00
======== xab
Date/Time Original              : 2015:08:12 17:39:48.479+01:00
======== xac
Date/Time Original              : 2015:08:12 17:39:48.545+01:00
======== xad
Date/Time Original              : 2015:08:12 17:39:48.611+01:00
======== xae
Date/Time Original              : 2015:08:12 17:39:48.677+01:00
======== xaf
Date/Time Original              : 2015:08:12 17:39:48.743+01:00
======== xag
Date/Time Original              : 2015:08:12 17:39:48.809+01:00
======== xah
Date/Time Original              : 2015:08:12 17:39:48.875+01:00
======== xai
Date/Time Original              : 2015:08:12 17:39:48.941+01:00
======== xaj
Date/Time Original              : 2015:08:12 17:39:49.007+01:00
======== xak
Date/Time Original              : 2015:08:12 17:39:49.073+01:00
======== xal
Date/Time Original              : 2015:08:12 17:39:49.139+01:00
======== xam
Date/Time Original              : 2015:08:12 17:39:49.205+01:00
======== xan
Date/Time Original              : 2015:08:12 17:39:49.271+01:00
======== xao
Date/Time Original              : 2015:08:12 17:39:49.337+01:00
======== xap
Date/Time Original              : 2015:08:12 17:39:49.403+01:00
or all values:

Code: [Select]

$ exiftool xaa
ExifTool Version Number         : 9.54
File Name                       : xaa
Directory                       : .
File Size                       : 154 kB
File Modification Date/Time     : 2015:08:12 18:16:45+02:00
File Access Date/Time           : 2015:08:12 18:27:35+02:00
File Inode Change Date/Time     : 2015:08:12 18:16:45+02:00
File Permissions                : rw-r--r--
File Type                       : FLIR
MIME Type                       : application/unknown
Emissivity                      : 0.95
Object Distance                 : 1.00 m
Reflected Apparent Temperature  : 20.0 C
Atmospheric Temperature         : -4.0 C
IR Window Temperature           : 20.0 C
IR Window Transmission          : 1.00
Relative Humidity               : 50.0 %
Planck R1                       : 14345.714
Planck B                        : 1347.7
Planck F                        : 1
Atmospheric Trans Alpha 1       : 0.006569
Atmospheric Trans Alpha 2       : 0.012620
Atmospheric Trans Beta 1        : -0.002276
Atmospheric Trans Beta 2        : -0.006670
Atmospheric Trans X             : 1.900000
Camera Temperature Range Max    : 120.0 C
Camera Temperature Range Min    : -20.0 C
Camera Model                    : FLIR E40
Camera Part Number              : 49001-2001
Camera Serial Number            : 490337633
Camera Software                 : 20.0.0
Lens Model                      : FOL18
Lens Part Number                :
Lens Serial Number              :
Field Of View                   : 25.0 deg
Filter Model                    :
Filter Part Number              :
Filter Serial Number            :
Planck O                        : -5711
Planck R2                       : 0.01234
Raw Value Median                : 18457
Raw Value Range                 : 780
Date/Time Original              : 2015:08:12 17:39:48.413+01:00
Focus Step Count                : 1526
Focus Distance                  : 5.3 m

[tmbinc]

Meh. Okay.

I'm on 2.3, so maybe it was added later. Could someone with a Flir Ex with 2.3 firmware give it a try? I wouldn't know what else it could be.

[tomas123]

Do you see differences in the usb announcements "VideoControl Interface Descriptor" between my firmware 1.19.8  and your firmware 2.3?

Code: [Select]

$ lsusb -v
Bus 001 Device 006: ID 09cb:1007
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 ?
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  idVendor           0x09cb
  idProduct          0x1007
  bcdDevice            0.00
  iManufacturer           7 FLIR Systems
  iProduct                8 FLIR Ex-Series
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          241
    bNumInterfaces          3
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass         14 Video
      bFunctionSubClass       3 Video Interface Collection
      bFunctionProtocol       0
      iFunction              10 FLIR USB Video
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      1 Video Control
      bInterfaceProtocol      0
      iInterface              2 FLIR Ex-Series
      VideoControl Interface Descriptor:
        bLength                13
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdUVC               1.00
        wTotalLength           85
        dwClockFrequency        0.001000MHz
        bInCollection           1
        baInterfaceNr( 0)       1
      VideoControl Interface Descriptor:
        bLength                18
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             1
        wTerminalType      0x0201 Camera Sensor
        bAssocTerminal          0
        iTerminal               0
        wObjectiveFocalLengthMin      0
        wObjectiveFocalLengthMax      0
        wOcularFocalLength            0
        bControlSize                  3
        bmControls           0x00020260
          Focus (Absolute)
          Focus (Relative)
          Zoom (Absolute)
          Focus, Auto
      VideoControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             3
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bSourceID               6
        iTerminal               0
      VideoControl Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      4 (SELECTOR_UNIT)
        bUnitID                 4
        bNrInPins               1
        baSource( 0)            1
        iSelector               0
      VideoControl Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      5 (PROCESSING_UNIT)
      Warning: Descriptor too short
        bUnitID                 5
        bSourceID               4
        wMaxMultiplier          0
        bControlSize            2
        bmControls     0x00000217
          Brightness
          Contrast
          Hue
          Sharpness
          Gain
        iProcessing             0
        bmVideoStandards     0x1b
          None
          NTSC - 525/60
          SECAM - 625/50
          NTSC - 625/50
      VideoControl Interface Descriptor:
        bLength                27
        bDescriptorType        36
        bDescriptorSubtype      6 (EXTENSION_UNIT)
        bUnitID                 6
        guidExtensionCode         {d41f59fa-5094-463a-b3bb-e7858a831fa3}
        bNumControl             4
        bNrPins                 1
        baSourceID( 0)          5
        bControlSize            2
        bmControls( 0)       0x0f
        bmControls( 1)       0x00
        iExtension              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              16
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      2 Video Streaming
      bInterfaceProtocol      0
      iInterface              0
      VideoStreaming Interface Descriptor:
        bLength                            14
        bDescriptorType                    36
        bDescriptorSubtype                  1 (INPUT_HEADER)
        bNumFormats                         1
        wTotalLength                       79
        bEndPointAddress                  130
        bmInfo                              0
        bTerminalLink                       3
        bStillCaptureMethod                 0
        bTriggerSupport                     0
        bTriggerUsage                       0
        bControlSize                        1
        bmaControls( 0)                    27
      VideoStreaming Interface Descriptor:
        bLength                            27
        bDescriptorType                    36
        bDescriptorSubtype                  4 (FORMAT_UNCOMPRESSED)
        bFormatIndex                        1
        bNumFrameDescriptors                1
        guidFormat                            {59555932-0000-1000-8000-00aa00389b71}
        bBitsPerPixel                      16
        bDefaultFrameIndex                  1
        bAspectRatioX                       0
        bAspectRatioY                       0
        bmInterlaceFlags                 0x00
          Interlaced stream or variable: No
          Fields per frame: 2 fields
          Field 1 first: No
          Field pattern: Field 1 only
          bCopyProtect                      0
      VideoStreaming Interface Descriptor:
        bLength                            38
        bDescriptorType                    36
        bDescriptorSubtype                  5 (FRAME_UNCOMPRESSED)
        bFrameIndex                         1
        bmCapabilities                   0x03
          Still image supported
          Fixed frame-rate
        wWidth                            320
        wHeight                           240
        dwMinBitRate                   912384
        dwMaxBitRate                   912384
        dwMaxVideoFrameBufferSize      153600
        dwDefaultFrameInterval        2666664
        bFrameIntervalType                  3
        dwFrameInterval( 0)            666666
        dwFrameInterval( 1)           1333332
        dwFrameInterval( 2)           2666664
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 ?
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0000
  (Bus Powered)

A few hundred lines of code later (simply record all frames with a max temp of higher than some threshold, re-mapping to 8-bit, then encoding as mp4)

I'm very interested in your Fuchsjagd code snippets 

[OrBy]

Since mine is 2.3.0 I spun up a Ubuntu 14.04.3 VM (First time every trying my hand at compiling anything) and got as far at the line listed above before I for the following error:

Code: [Select]

orby@ubuntu:~/libuvc/build$ /usr/bin/cc  -o flir ../src/flir.c -luvc
../src/flir.c: In function ‘cb’:
../src/flir.c:30:3: warning: format ‘%d’ expects argument of type ‘int’, but argument 2 has type ‘__time_t’ [-Wformat=]
   printf("got frame %d:%d, %d bytes\n", tv.tv_sec, tv.tv_usec, frame->data_bytes);
   ^
../src/flir.c:30:3: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘__suseconds_t’ [-Wformat=]
../src/flir.c:30:3: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has type ‘size_t’ [-Wformat=]

Everything seemed to go word for word as outlined in tomas123 detailed instructions till that point.

I have no issues saying that I am in way over my head so unless anyone has any suggestions I don't think I can test further.

I would love to get a libuvc with this patch with this working on windows but my limited looking into it seems to indicate that the USB driver wrangling situation to accomplish this would be even more complex.

[Bud]

the result is the known screen image and not a RAW image:(

Same here, with FW 2.3.0

Code: [Select]

root@Ubuntu32:/home/some/Temp/libuvc# ./myflir test6.raw
UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x240
  bit rate: 912384-912384
  max frame size: 153600
  default interval: 1/3
  interval[0]: 1/15
  interval[1]: 1/7
  interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming...
got frame 1439442632:196459, 153600 bytes
got frame 1439442632:290426, 153600 bytes
got frame 1439442632:384196, 153600 bytes
got frame 1439442632:478789, 153600 bytes
got frame 1439442632:573349, 153600 bytes

Done streaming.
Device closed
UVC exited


An extracted frame then is just a screen capture.

[tomas123]

Since mine is 2.3.0 I spun up a Ubuntu 14.04.3 VM (First time every trying my hand at compiling anything) and got as far at the line listed above before I for the following error:

Code: [Select]

orby@ubuntu:~/libuvc/build$ /usr/bin/cc  -o flir ../src/flir.c -luvc
../src/flir.c: In function ‘cb’:
../src/flir.c:30:3: warning: format ‘%d’ expects argument of type ‘int’, but argument 2 has type ‘__time_t’ [-Wformat=]
   printf("got frame %d:%d, %d bytes\n", tv.tv_sec, tv.tv_usec, frame->data_bytes);
   ^
../src/flir.c:30:3: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘__suseconds_t’ [-Wformat=]
../src/flir.c:30:3: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has type ‘size_t’ [-Wformat=]

Everything seemed to go word for word as outlined in tomas123 detailed instructions till that point.

I have no issues saying that I am in way over my head so unless anyone has any suggestions I don't think I can test further.

I would love to get a libuvc with this patch with this working on windows but my limited looking into it seems to indicate that the USB driver wrangling situation to accomplish this would be even more complex.

This is only a compiler warning, not an error. I got the excact same warning.
Simple use the successful compiled flir binary.

[tmbinc]

I don't have my E4 here right now. I'll investigate. Sorry for wasting everyone's time. (It definitely worked for me, as you can see in the youtube video)

[tomas123]

Sorry, but I'm don't understand the "bFormatIndex - 1" hack.
Can you give us some more informations about the patch?

Which stream from the lsusb tree (see above) we are grabbing?

Thanks for help!

Code: [Select]

diff --git a/src/stream.c b/src/stream.c
index 571be16..aabb6ff 100644
--- a/src/stream.c
+++ b/src/stream.c
@@ -183,6 +183,7 @@ uvc_error_t uvc_query_stream_ctrl(
   /* prepare for a SET transfer */
   if (req == UVC_SET_CUR) {
     SHORT_TO_SW(ctrl->bmHint, buf);
+    ctrl->bFormatIndex = 2;
     buf[2] = ctrl->bFormatIndex;
     buf[3] = ctrl->bFrameIndex;
     INT_TO_DW(ctrl->dwFrameInterval, buf + 4);
@@ -222,6 +223,7 @@ uvc_error_t uvc_query_stream_ctrl(
   if (req != UVC_SET_CUR) {
     ctrl->bmHint = SW_TO_SHORT(buf);
     ctrl->bFormatIndex = buf[2];
+    ctrl->bFormatIndex = 1;
     ctrl->bFrameIndex = buf[3];
     ctrl->dwFrameInterval = DW_TO_INT(buf + 4);
     ctrl->wKeyFrameRate = SW_TO_SHORT(buf + 8);
@@ -828,7 +830,7 @@ uvc_error_t uvc_stream_start(
   strmh->pts = 0;
   strmh->last_scr = 0;
 
-  frame_desc = uvc_find_frame_desc_stream(strmh, ctrl->bFormatIndex, ctrl->bFrameIndex);
+  frame_desc = uvc_find_frame_desc_stream(strmh, ctrl->bFormatIndex  - 1, ctrl->bFrameIndex);
   if (!frame_desc) {
     ret = UVC_ERROR_INVALID_PARAM;
     goto fail;
@@ -1037,7 +1039,7 @@ void _uvc_populate_frame(uvc_stream_handle_t *strmh) {
    * is going to be reopen_on_change anyway
    */
 
-  frame_desc = uvc_find_frame_desc(strmh->devh, strmh->cur_ctrl.bFormatIndex,
+  frame_desc = uvc_find_frame_desc(strmh->devh, strmh->cur_ctrl.bFormatIndex - 1,
     strmh->cur_ctrl.bFrameIndex);
 
   frame->frame_format = strmh->frame_format;

[tomas123]

I think, the RAW Video of E40 is

Code: [Select]

bFormatIndex= 2
dwMaxVideoFrameBufferSize = 157440
but the flir.c log:

Code: [Select]

bFormatIndex: 1
bFrameIndex: 1
dwMaxVideoFrameSize: 157440
...very magic

compress output from lsusb -v

E40

      VideoStreaming Interface Descriptor:
        bFormatIndex                        1
        bBitsPerPixel                      16
        wWidth                            320
        wHeight                           240
        dwMaxVideoFrameBufferSize      153600 // = 320x240x2 Byte ( Screen Shot Video)

      
     VideoStreaming Interface Descriptor:
        bFormatIndex                        2
        bBitsPerPixel                      16
        wWidth                            320
        wHeight                           246    // 6 extra lines for the FFF Header (see exiftool output above)
        dwMaxVideoFrameBufferSize      157440 // 157440 is same size as the output from the RAW Video grab tool
 
 
 E4
 
      VideoStreaming Interface Descriptor:
        bFormatIndex                        1
        bBitsPerPixel                      16
        wWidth                            320
        wHeight                           240
        dwMaxVideoFrameBufferSize      153600 // = 320x240x2 Byte ( Screen Shot Video)

[OrBy]

This is only a compiler warning, not an error. I got the excact same warning.
Simple use the successful compiled flir binary.

I will spin that VM back up and test it out when I get off work later today then.

I don't have my E4 here right now. I'll investigate. Sorry for wasting everyone's time. (It definitely worked for me, as you can see in the youtube video)

I don't think anyone thinks you have wasted their time. (I don't) There could be different configurations or variables that could be at play here.

What USB mode is your camera in when capturing? (just UVC or one of the mixed modes?)

[tomas123]

What USB mode is your camera in when capturing? (just UVC or one of the mixed modes?)

I checked all UVC variants without success.

[OrBy]

Just took it for a little spin but getting some odd results:

Code: [Select]

orby@ubuntu:~$ sudo libuvc/build/flir e4-1.raw
UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x240
  bit rate: 912384-912384
  max frame size: 153600
  default interval: 1/3
  interval[0]: 1/15
  interval[1]: 1/7
  interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming...
got frame 1439507727:121939, 153600 bytes
got frame 1439507727:122047, 153600 bytes
orby@ubuntu:~$ split -b 153600 e4-1.raw
orby@ubuntu:~$ convert -size 320x240  yuv:xaa -separate 4.png
convert.im6: unexpected end-of-file `xaa': No such file or directory @ error/yuv.c/ReadYUVImage/428.

It seems like it may be the raw feed at least for me.
Attached are the 2 png's it spat out and the zip has the raw and a xaa file it in it.

[Bud]

Meh. Okay.

I'm on 2.3, so maybe it was added later. Could someone with a Flir Ex with 2.3 firmware give it a try? I wouldn't know what else it could be.

Can you share your patched stream.c ?

[Bud]

@OrBy

I am getting same error and same result (Ubuntu 11.10 on a VM). But the png produced is not radiometric. Yours does not seem to be it either, it does not contain radiometric metadata.

[tomas123]

@Orby

your png-image is radiometric with swapped bytes
see my tutorial here: https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg348398/#msg348398

Code: [Select]

>identify 4-0.png
4-0.png PNG 320x240 320x240+0+0 16-bit sRGB 67.3KB 0.000u 0:00.001
>convert -define png:swap-bytes=on 4-0.png -auto-level raw.png
better use the splitted part

Code: [Select]

convert -depth 16 -size 320x240 -auto-level gray:xaa raw.png
@Orby + @Bud
You have both the firmware 2.3.0.
Find the difference between the configuration 

check "lsusb -v"

our logs from flir.c are 100% identical

Code: [Select]

UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x240
  bit rate: 912384-912384
  max frame size: 153600
  default interval: 1/3
  interval[0]: 1/15
  interval[1]: 1/7
  interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming...
A interesting difference to my E40 stream:
The E4 RAW stream contains no FFF Header:
Image Size: 320x240x2 Byte =  frame size =  153600 Byte

The E40 RAW Stream contains full FFF Imgages
frame size 157440 byte
see sample in my post https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg729688/#msg729688

[OrBy]

@Orby

your png-image is radiometric with swapped bytes
see my tutorial here: https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg348398/#msg348398

Code: [Select]

>identify 4-0.png
4-0.png PNG 320x240 320x240+0+0 16-bit sRGB 67.3KB 0.000u 0:00.001
>convert -define png:swap-bytes=on 4-0.png -auto-level raw.png
better use the splitted part

Code: [Select]

convert -depth 16 -size 320x240 -auto-level gray:xaa raw.png
@Orby + @Bud
You have both the firmware 2.3.0.
Find the difference between the configuration 

check "lsusb -v"

our logs from flir.c are 100% identical

Code: [Select]

UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x240
  bit rate: 912384-912384
  max frame size: 153600
  default interval: 1/3
  interval[0]: 1/15
  interval[1]: 1/7
  interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming...
A interesting difference to my E40 stream:
The E4 RAW stream contains no FFF Header:
Image Size: 320x240x2 Byte =  frame size =  153600 Byte

The E40 RAW Stream contains full FFF Imgages
frame size 157440 byte
see sample in my post https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg729688/#msg729688

Yes I kinda thought that it had the look like it was backwards and that it needed just a bit more processing to look right. I ran out of time to play last night but I was going to try messing with it a little more when time permitted.

It's interesting how the E4 is missing the header info - guess they cant make it too easy!

I will post up my lsusb -v log when I get some time back on my PC.

[tomas123]

The only missing header infos are the time stamps.

All other Flir values to calculate the real temperature of a pixel are (calibrated) constants.
You can get them from every radiometric jpg of the same (!) camera.

I hope tmbinc give us some explanations to the bFormatIndex-hack
and some code snippets for our own Fuchsjagd   

[Bud]

By non-radiometric i mean no temperature scaling information is in the file. I do not have access to my computer at the moment but if you load the image in Flir Tools i think temperature measurements will be incorrect and all the way between -40 +300 or something. To get it right i think the image levels need to be scaled against a known reference, same as you do when creating panoramas after stitching step.
Anyway, try loading it in Flir Tools to see if it can be correctly measured.

[tomas123]

@Bud
Currently you have not radiometric raw values in your stream.  Therefore Flir Tools can't read anything.

You need a RAW stream, like Orby E4.
Then you can inject a extracted 16 bit raw png in a radiometric jpg (from the same E4!) and the temperature measurement of Flir Tools works fine

simple use my (panorama) script splitjpg.php for injecting
https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg348715/#msg348715

// no -auto-level 

Code: [Select]

$ convert -depth 16 -size 320x240  gray:xaa raw.png
$ php splitjpg.php -i IR_0xxx.jpg -r raw.png -o sampleLoad the sample.jpg in Flir Tools

[Bud]

Yes i have already tried that with my non- radiometric streamed images and if i use your splitjpg script and a reference radiometric jpg, i get a proper thermal image in Flir Tools but temperature readings are still somewhat off, probably because the reference jpg was not from the same scene. I will try tonight without the scaling step.

Having said that, i begin to believe that may not be difference between the E4 hacked radiometric streamed images vs screen capture streamed ones, providing i turn measurements and everything else off on the camera, and also turn MSX off so the screen only displays the thermal image. I can do this on my doctored E4 2.3.0.

 I will still try to get the camera stream same way as OrBy's so i can post processes images received using the two methods and compare the end result.

[OrBy]

I will still try to get the camera stream same way as OrBy's so i can post processes images received using the two methods and compare the end result.

I can grab some more out of each stream and put them up for comparison later today if needed/wanted.

[Bud]

Having said that, i begin to believe that may not be difference between the E4 hacked radiometric streamed images vs screen capture streamed ones

...except perhaps swapped bytes in radiometric stream.

[tomas123]

i begin to believe that may not be difference between the E4 hacked radiometric streamed images vs screen capture streamed ones, providing i turn measurements and everything else off on the camera, and also turn MSX off so the screen only displays the thermal image.

Stream of the monitor view is nice but a raw stream is really cool (surveillance camera, measurements)
See the great video from tmbinc.
I hope that tmbinc give us the code for converting a raw stream. I see a nice application for my raspberry pi.

Don't forget that Flir Tools can grab a raw stream from E40 (switch from video to signal)

https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg377140/#msg377140

the video stream of E4 grab with the free FLIR IR Camera Player

https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg378926/#msg378926

[tomas123]

@OrBy

Your upgraded the E4 from 1.18.7 to 2.3.0.
Interesting!
Maybe the (hidden) raw stream of your E4 is activated by a switch in your config file.

Just wanted to give a quick  to janekivi and everyone else that's been working on 2.3.0.

I decided to upgrade from 1.18.7 to 2.3.0 today on my 1.0 E4.

I just flashed the 2.3.0 FIF and rebooted and it retained my "upgraded" status. Installed the 2014 menu FIF and had to add a little tweak to my e8.cfg and recrc it to get the manual adjustments working.

So far I really like the clean no measurements screen! Makes for "truer" video capture in UVC mode and allows me to "frame" my shots better! Also the thermal blending is a nice addition.

Cant wait till I have some more time to play with the advanced ed.

[OrBy]

Yes I did upgrade mine to 2.3.0

Here is my lsusb -v dump:
(I hope it's the right section required)

Code: [Select]

Bus 001 Device 002: ID 09cb:1007 
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 ?
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  idVendor           0x09cb
  idProduct          0x1007
  bcdDevice            0.00
  iManufacturer           7
  iProduct                8
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          241
    bNumInterfaces          3
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass         14 Video
      bFunctionSubClass       3 Video Interface Collection
      bFunctionProtocol       0
      iFunction              10
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      1 Video Control
      bInterfaceProtocol      0
      iInterface              2
      VideoControl Interface Descriptor:
        bLength                13
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdUVC               1.00
        wTotalLength           85
        dwClockFrequency        0.001000MHz
        bInCollection           1
        baInterfaceNr( 0)       1
      VideoControl Interface Descriptor:
        bLength                18
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             1
        wTerminalType      0x0201 Camera Sensor
        bAssocTerminal          0
        iTerminal               0
        wObjectiveFocalLengthMin      0
        wObjectiveFocalLengthMax      0
        wOcularFocalLength            0
        bControlSize                  3
        bmControls           0x00020260
          Focus (Absolute)
          Focus (Relative)
          Zoom (Absolute)
          Focus, Auto
      VideoControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             3
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bSourceID               6
        iTerminal               0
      VideoControl Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      4 (SELECTOR_UNIT)
        bUnitID                 4
        bNrInPins               1
        baSource( 0)            1
        iSelector               0
      VideoControl Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      5 (PROCESSING_UNIT)
      Warning: Descriptor too short
        bUnitID                 5
        bSourceID               4
        wMaxMultiplier          0
        bControlSize            2
        bmControls     0x00000217
          Brightness
          Contrast
          Hue
          Sharpness
          Gain
        iProcessing             0
        bmVideoStandards     0x1b
          None
          NTSC - 525/60
          SECAM - 625/50
          NTSC - 625/50
      VideoControl Interface Descriptor:
        bLength                27
        bDescriptorType        36
        bDescriptorSubtype      6 (EXTENSION_UNIT)
        bUnitID                 6
        guidExtensionCode         {d41f59fa-5094-463a-b3bb-e7858a831fa3}
        bNumControl             4
        bNrPins                 1
        baSourceID( 0)          5
        bControlSize            2
        bmControls( 0)       0x0f
        bmControls( 1)       0x00
        iExtension              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              16
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 Video
      bInterfaceSubClass      2 Video Streaming
      bInterfaceProtocol      0
      iInterface              0
      VideoStreaming Interface Descriptor:
        bLength                            14
        bDescriptorType                    36
        bDescriptorSubtype                  1 (INPUT_HEADER)
        bNumFormats                         1
        wTotalLength                       79
        bEndPointAddress                  130
        bmInfo                              0
        bTerminalLink                       3
        bStillCaptureMethod                 0
        bTriggerSupport                     0
        bTriggerUsage                       0
        bControlSize                        1
        bmaControls( 0)                    27
      VideoStreaming Interface Descriptor:
        bLength                            27
        bDescriptorType                    36
        bDescriptorSubtype                  4 (FORMAT_UNCOMPRESSED)
        bFormatIndex                        1
        bNumFrameDescriptors                1
        guidFormat                            {59555932-0000-1000-8000-00aa00389b71}
        bBitsPerPixel                      16
        bDefaultFrameIndex                  1
        bAspectRatioX                       0
        bAspectRatioY                       0
        bmInterlaceFlags                 0x00
          Interlaced stream or variable: No
          Fields per frame: 2 fields
          Field 1 first: No
          Field pattern: Field 1 only
          bCopyProtect                      0
      VideoStreaming Interface Descriptor:
        bLength                            38
        bDescriptorType                    36
        bDescriptorSubtype                  5 (FRAME_UNCOMPRESSED)
        bFrameIndex                         1
        bmCapabilities                   0x03
          Still image supported
          Fixed frame-rate
        wWidth                            320
        wHeight                           240
        dwMinBitRate                   912384
        dwMaxBitRate                   912384
        dwMaxVideoFrameBufferSize      153600
        dwDefaultFrameInterval        2666664
        bFrameIntervalType                  3
        dwFrameInterval( 0)            666666
        dwFrameInterval( 1)           1333332
        dwFrameInterval( 2)           2666664
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0

And here are the contents of my e8.cfg:

Code: [Select]

.caps entry
.caps.config entry
.caps.config.name text "app E8"
.caps.config.revision text "1.0"
.caps.config.image entry
.caps.config.image.framegrab entry
.caps.config.image.framegrab.fusion entry
.caps.config.image.framegrab.fusion.enabled bool true
.caps.config.image.framegrab.fusion.pip entry
.caps.config.image.framegrab.fusion.pip.enabled bool true
.caps.config.image.framegrab.fusion.hcf entry
.caps.config.image.framegrab.fusion.hcf.enabled bool true
.caps.config.image.services entry
.caps.config.image.services.store entry
.caps.config.image.services.store.enabled bool true
.caps.config.image.services.store.radiometric entry
.caps.config.image.services.store.radiometric.enabled bool true
.caps.config.image.services.store.incompatible entry
.caps.config.image.services.store.incompatible.enabled bool false
.caps.config.image.services.store.incompatible.level int32 0
.caps.config.image.settings entry
.caps.config.image.settings.enabled bool true
.caps.config.image.settings.IRwidth int32 320
.caps.config.image.settings.IRheight int32 240
.caps.config.image.sysimg entry
.caps.config.image.sysimg.alarms entry
.caps.config.image.sysimg.alarms.enabled bool true
.caps.config.image.sysimg.alarms.measfunc entry
.caps.config.image.sysimg.alarms.measfunc.enabled bool true
.caps.config.image.sysimg.alarms.measfunc.maxCount int32 3
.caps.config.image.sysimg.alarms.humidity entry
.caps.config.image.sysimg.alarms.humidity.enabled bool true
.caps.config.image.sysimg.alarms.humidity.maxCount int32 1
.caps.config.image.sysimg.alarms.insulation entry
.caps.config.image.sysimg.alarms.insulation.enabled bool true
.caps.config.image.sysimg.alarms.insulation.maxCount int32 1
.caps.config.image.sysimg.irMarkers entry
.caps.config.image.sysimg.irMarkers.enabled bool true
.caps.config.image.sysimg.irMarkers.spot entry
.caps.config.image.sysimg.irMarkers.spot.enabled bool false
.caps.config.image.sysimg.irMarkers.spot.maxCount int32 0
.caps.config.image.sysimg.irMarkers.arrow entry
.caps.config.image.sysimg.irMarkers.arrow.enabled bool true
.caps.config.image.sysimg.irMarkers.arrow.maxCount int32 4
.caps.config.image.sysimg.irMarkers.box entry
.caps.config.image.sysimg.irMarkers.box.enabled bool false
.caps.config.image.sysimg.irMarkers.box.maxCount int32 0
.caps.config.image.sysimg.measureFuncs entry
.caps.config.image.sysimg.measureFuncs.enabled bool true
.caps.config.image.sysimg.measureFuncs.diff entry
.caps.config.image.sysimg.measureFuncs.diff.enabled bool true
.caps.config.image.sysimg.measureFuncs.diff.maxCount int32 1
.caps.config.image.sysimg.measureFuncs.diff.calcMask int32 65526
.caps.config.image.sysimg.measureFuncs.isotherm entry
.caps.config.image.sysimg.measureFuncs.isotherm.enabled bool true
.caps.config.image.sysimg.measureFuncs.isotherm.calcMask int32 20
.caps.config.image.sysimg.measureFuncs.isotherm.dual bool false
.caps.config.image.sysimg.measureFuncs.isotherm.fixScale bool false
.caps.config.image.sysimg.measureFuncs.isotherm.interval bool true
.caps.config.image.sysimg.measureFuncs.isotherm.invInterval bool false
.caps.config.image.sysimg.measureFuncs.isotherm.maxCount int32 1
.caps.config.image.sysimg.measureFuncs.mbox entry
.caps.config.image.sysimg.measureFuncs.mbox.enabled bool true
.caps.config.image.sysimg.measureFuncs.mbox.calcMask int32 1924
.caps.config.image.sysimg.measureFuncs.mbox.maxCount int32 5
.caps.config.image.sysimg.measureFuncs.mcircle entry
.caps.config.image.sysimg.measureFuncs.mcircle.enabled bool false
.caps.config.image.sysimg.measureFuncs.mcircle.calcMask int32 1924
.caps.config.image.sysimg.measureFuncs.mcircle.maxCount int32 0
.caps.config.image.sysimg.measureFuncs.mline entry
.caps.config.image.sysimg.measureFuncs.mline.enabled bool false
.caps.config.image.sysimg.measureFuncs.mline.calcMask int32 1924
.caps.config.image.sysimg.measureFuncs.mline.maxCount int32 0
.caps.config.image.sysimg.measureFuncs.reftemp entry
.caps.config.image.sysimg.measureFuncs.reftemp.enabled bool true
.caps.config.image.sysimg.measureFuncs.reftemp.calcMask int32 1924
.caps.config.image.sysimg.measureFuncs.reftemp.maxCount int32 1
.caps.config.image.sysimg.measureFuncs.script entry
.caps.config.image.sysimg.measureFuncs.script.enabled false
.caps.config.image.sysimg.measureFuncs.script.maxCount int32 0
.caps.config.image.sysimg.measureFuncs.spot entry
.caps.config.image.sysimg.measureFuncs.spot.enabled bool true
.caps.config.image.sysimg.measureFuncs.spot.calcMask int32 514
.caps.config.image.sysimg.measureFuncs.spot.maxCount int32 5
.caps.config.image.sysimg.visualMarkers entry
.caps.config.image.sysimg.visualMarkers.enabled bool true
.caps.config.image.sysimg.visualMarkers.spot entry
.caps.config.image.sysimg.visualMarkers.spot.enabled bool false
.caps.config.image.sysimg.visualMarkers.spot.maxCount int32 0
.caps.config.image.sysimg.visualMarkers.arrow entry
.caps.config.image.sysimg.visualMarkers.arrow.enabled bool true
.caps.config.image.sysimg.visualMarkers.arrow.maxCount int32 4
.caps.config.image.sysimg.visualMarkers.box entry
.caps.config.image.sysimg.visualMarkers.box.enabled bool false
.caps.config.image.sysimg.visualMarkers.box.maxCount int32 0
.caps.config.image.contadj entry
.caps.config.image.contadj.minSpanFactor entry
.caps.config.image.contadj.minSpanFactor.enabled bool true
.caps.config.image.contadj.minSpanFactor.factorAuto double 2.0
.caps.config.image.contadj.minSpanFactor.factorManual double 2.0
.caps.config.image.targetNoise entry
.caps.config.image.targetNoise.enabled bool false
.caps.config.image.targetNoise.targetNoiseMk int32 0
.caps.config.image.zoom entry
.caps.config.image.zoom.enabled bool true
.caps.config.image.zoom.maxFactor double 8
.caps.config.system entry
.caps.config.system.focus entry
.caps.config.system.focus.laser entry
.caps.config.system.focus.laser.updateFocus entry
.caps.config.system.focus.laser.updateFocus.enabled bool true
.caps.config.ui.image entry
.caps.config.ui.image.adjust entry
.caps.config.ui.image.adjust.enabled bool true
.caps.config.ui.image.adjust.manual bool true
.caps.config.ui entry
.caps.config.ui.fusion entry
.caps.config.ui.fusion.PIP entry
.caps.config.ui.fusion.PIP.enabled bool true
.caps.hw entry
.caps.hw.sdcard entry
.caps.hw.sdcard.enabled bool false


[Bud]

I followed tomas123's steps to re-compile libuvc library and now it works for me too (thanks tomas123 for posting the compilation log, I think I can learn Chinese faster than Linux ).

Attached is a frame from the video stream, loaded in FLIR Tools and some measurements enabled. All makes sense and matches what is measured by the camera. For the record, the camera is a native E4 fw 2.3.0 doctored for resolution and enhanced menu.

So yes the patch works, it is  possible to stream radiometric video from E4 (oh well, for now just in a file and using Linux), thanks user tmbinc 

[tomas123]

Great!
Please describe all steps, how you can load a frame in Flir Tools and attach a splitted single raw frame (as .zip)

I think, I must make a firmware update from 1.19 to 2.3.0
don't touch a running system 

[Bud]

All as previously described:

Pre-requisits for image processing software:

- imagemagick
- php
- splitjpg.php script
    source: https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg348715/#msg348715
- a sample image from the same E4 camera
- FLIR Tools

workflow:

- capture a RAW video
- split to individual frames

Code: [Select]

split -b 153600 test1.raw
- convert the required frame (in this example the first frame) to a RAW png

Code: [Select]

convert -depth 16 -size 320x240  gray:xaa raw1.png
convert the RAW png to radiometric jpg (for a template use a sample jpg image, e.g. FLIR0123.jpg, from the same camera:

Code: [Select]

php splitjpg.php -i FLIR0123.jpg -r raw1.png -o test1.jpgnote: after this step the resulting test1.jpg preview will appear the same as the template image, do not worry: the actual RAW data is hidden behind the image.

Now load this JPG into FLIR Tools to rebuild the preview image, place temperature measurements, apply a desired palette and resave.

I do video capture, split and convert on a Ubuntu VM, everything else on a Windows 7.

[Bud]

Found a great little utility called ImageJ

http://imagej.nih.gov/ij/

Holy sh!t, have not seen it before, it is a jewel. For the purpose of this thread it can directly open raw files created by tmbinc's software and resave in a bunch of different formats including individual split frames. Also has a pile of built-in pallettes (Look Up Table they call it) and can colorize the video and even save the video in AVI format ready to be played ! 

A sample is attached - a radiometric stream capture from the E4, colorized using ImageJ.

test13.avi  <- click here

With this sort of post processing software radiometric streaming from E4 all of a sudden makes sense.

[tomas123]

  Great!!!
ImageJ read my E40 raw stream with little endian byte order and 3840 Byte Flir-FFF-Header between frames

@Bud
I linked your last two posts in my E4 link list:
https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg342072/#msg342072
see topic 15

Mike linked my post to the the head of the infinite thread
https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/
"Tomas123's links on post- processing and measurements using E4 images"


Please add in your second last post a link to my splitjpg.php source
https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg348715/#msg348715

@tmbinc
I hope you can give us your Fuchsjagd code snippets 

[Bud]

@Bud
Please add in your second last post a link to my splitjpg.php source
https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg348715/#msg348715

Done.

I wish ImageJ could combine multiple frames in a single image for superresolution, but have not figured out yet if it can.
A newer version is here:

http://fiji.sc/Downloads

Better palettes, +tons of image manipulation menus, a holy grail of image processing 

[tomas123]

a little bit off topic: superresolution with RAW thermal video

I think, you know my posts to superresolution.
(see my link list #17 -> https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg363688/#msg363688)

I have had good experiences with:

- Registax (sharp with wavelets)

- AviStack  (sharp with wavelets)

- Photoshop CS3 -> automate -> merge to HDR (automat. align images)
  -> quality like Registax/AviStax (CS3 works better as CS5)

... do not expect too much and resize your images/video before stacking

[Bud]

I wish ImageJ could combine multiple frames in a single image for superresolution, but have not figured out yet if it can.
A newer version is here:

http://fiji.sc/Downloads

Better palettes, +tons of image manipulation menus, a holy grail of image processing 

OK I think there is a few ways to experiment with superresolution in ImageJ. For the purpose of this topic the workflow to create a combined image from a sequence of video frames may look as follows:

- capture a few seconds of a raw video as already described above
- open the video in ImageJ (File->Import->RAW)
- if necessary, delete bad or unwanted frames (use scroll bar to select a frame, then "Image->Stacks->Delete Slice"
- combine the frames (Image->Stacks->Z Project)

[tmbinc]

Sorry for not keeping up with this thread.

The idea is that even though the E4 doesn't announce the bFormatIndex=2 RAW stream via the USB descriptor, it still allows to select it. The code that restricts the raw mode based on the camera type does cripple the descriptor, but it doesn't prevent you from still selecting it.

The libuvc patch is what I've posted (i just double-checked against what's still on my raspi2), I applied it on top of https://github.com/ktossell/libuvc/commits/6ff39f17e3f3a26f437bc6b3c09cc6c289db7891 . It does this in a very lame and stupid way by switching it back and forth, to make the library take the original stream descriptor, but send the right UVC command.

I attached the "fuchsjags" source, but it's really not great - it calculates min and max, scales that to 0..255, inserts the texts and the pointer, converts it to ARGB for the framebuffer, and also dumps the raw and converted values into files. I have a different script that used avconv to encode it to MP4 ("avconv -f rawvideo -pix_fmt gray -s:v 320x240 -r 9 -i /mnt/fox/r1_proc.raw -c:v libx264 /mnt/fox/output.mp4").

[tomas123]

Thank You 

My next project name is Marderjagd 

[tomas123]

I think, I must make a firmware update from 1.19.8 to 2.3.0
don't touch a running system 

successful upgraded the E4 from 1.19.8 to 2.3.0 with Flir Tools
The resolution hack survived the update (nothing else to do).

the RAW stream grabbing now works fine

[Bud]

Great, so RAW streaming did not work in 1.19.8 ...

[Bud]

@tomas123

I wonder if you then could take a challenge and compile the other sample from libuvc which supposedly creates a window and displays the streamed video (source modified for 320x240x15 is attached). It requires a library called opencv to be installed. I tried but as I said my Chinese is much better than my Linux, no matter how hard I tried I got the dreaded opencv "Rebuild the library with Windows, GTK+ 2.x or Carbon support" error. I am sure I am missing something obvious but I just do not know what.

If this code works and if combined with tmbinc's code this may become a usable program for E4 that displays, saves and processes E4 radiometric video.

[tmbinc]

By the way, my test tool already shows the output on a framebuffer. It's not great and was just meant for debugging... Should be easy though to add a LUT and some color scheme, as well as "calibrated" temperature. I'm not sure though that the radiometric metadata is available.

I also tried doing a 2D-FFT to assist in fine-tuning focus. But the results haven't been that useful. Does someone have a better idea how to do this? (I.e. how to evaluate focus based on image processing)

[Zucca]

l upgraded the E4 from 1.19.8 to 2.3.0 with Flir Tools
The resolution hack survived the update (nothing else to do).

Temped to do the same a long time ago, didn´t have time yet:

https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg607838/#msg607838

and do the janekivi menu mod. Hope the raw capturing will work regardless the menu mod.

[Bud]

I also tried doing a 2D-FFT to assist in fine-tuning focus. But the results haven't been that useful. Does someone have a better idea how to do this? (I.e. how to evaluate focus based on image processing)

The opencv library is a tool for computer vision, so I'd think it may have something in it for what you need.

[tomas123]

@tomas123

I wonder if you then could take a challenge and compile the other sample from libuvc which supposedly creates a window and displays the streamed video (source modified for 320x240x15 is attached). It requires a library called opencv to be installed. I tried but as I said my Chinese is much better than my Linux, no matter how hard I tried I got the dreaded opencv "Rebuild the library with Windows, GTK+ 2.x or Carbon support" error. I am sure I am missing something obvious but I just do not know what.

If this code works and if combined with tmbinc's code this may become a usable program for E4 that displays, saves and processes E4 radiometric video.

Hi Bud,

your code snippet works fine, no changes are necessary 

edit 30.08.15: found the source https://github.com/ktossell/libuvc/blob/master/src/test.c   

the steps:

(1) check version

Code: [Select]

$ lsb_release -a
Description: Ubuntu 14.04.1 LTS

(2) compile opencv
use this tutorial
http://www.bogotobogo.com/OpenCV/opencv_3_tutorial_ubuntu14_install_cmake.php

Code: [Select]

$ mkdir OpenCV
$ cd OpenCV
$ git clone [url]https://github.com/Itseez/opencv.git[/url]
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON ..
$ make
$ sudo make install
$ sudo ldconfig

(3) compile your sample
// ignore warnings
// note http://stackoverflow.com/questions/9614872/opencv-application-not-library-linking-errors

Code: [Select]

$ /usr/bin/cc  -o mytest mytest.c -luvc `pkg-config --cflags --libs opencv`
mytest.c: In function ‘cb’:
mytest.c:50:66: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
   printf("callback! length = %u, ptr = %d\n", frame->data_bytes, (int) ptr);
                                                                  ^
mytest.c:50:3: warning: format ‘%u’ expects argument of type ‘unsigned int’, but argument 2 has type ‘size_t’ [-Wformat=]
   printf("callback! length = %u, ptr = %d\n", frame->data_bytes, (int) ptr);

(4) run  your code

Code: [Select]

$ sudo ./mytest
UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x240
  bit rate: 912384-912384
  max frame size: 153600
  default interval: 1/3
  interval[0]: 1/15
  interval[1]: 1/7
  interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming for 10 seconds...
set_ae_mode: Pipe (-9)
callback! length = 153600, ptr = -886544208
set_exp_abs: Pipe (-9)
init done
opengl support available
callback! length = 157690, ptr = -886544208
callback! length = 157690, ptr = -886544208
callback! length = 157690, ptr = -886544208
callback! length = 157690, ptr = -886544208

and you get a window with a nice false color video 

[Ben321]

[Apologize if this is common knowledge and I just missed the memo. As everyone knows, the main thread is _huge_ and there was some talk about this, but I didn't find the final verdict of whether it's possible or not.]

I was trying to catch a fox that ate our pet rabbits from our garden on camera (people say they return to the scene of the crime). I wasn't successful (yet!), but I learned to hate the FLIR UVC implementation since it's randomly stops working during the night.

In an attempt to rectify the situation, I looked at the UVC implementation on the camera, and figured out that there's a dormant function to stream 9Hz raw radiometric data instead of just capturing the screen. With Windows and the regular UVC drivers, the best I could capture was the screen with 3.3Hz (for whatever reason; 15Hz should be possible) at (effectively) 8-bit over a pre-selected temp range.

The catch is that the UVC driver on the Flir explicitely disables radiometric streaming for the Z3 (=Flir Ex). It enables it for other cameras, though - i think the Flir Ex0 suports it out-of-the-box. The UVC stack is implemented in usbfnvideo.dll, which unfortunately is part of nk.bin, so it can't be easily patched (at least, I can't with my limited WinCE knowledge and my unwillingness to brick the E4).

The great thing is that FLIR sucks at implementing device restrictions; they have a dedicated grabber thread for the Flir Ex-series, and it _does_ indeed implement grabbing the raw data (via FVD1: instead of capturing the screen), and they _let_ you select capturing that (via setting bFormatIndex in the stream control), but they don't announce the ability for that in the UVC descriptors, so any "well-behaving" UVC client will not allow you to select it.

I've hacked libuvc (which is based on libusb) and - tada, it just worked. I patched libuvc pretty crudely so I'm sure there's a better way to do it. I've attached the patch and the crude sample (which just dumps out raw data into a file).

Let me know if you find this useful.

Cool, but could you please attach the compiled file instead of the C source code and patch? I want a binary to use directly. I don't have much experience with C programming, but I really would like to get to use this (I'm considering getting a Flir E4, part of the Flir Ex series, but whether or not I get it will depend on what features I can access, via EASY to use hacks, and having to compile a hack is NOT easy). So can you pleas compile this, and then post the compiled binary file here?

[tomas123]

this a generic linux source code and you must compile it for your CPU (ARM / Intel etc) and your Linux version
tmbinc compiled the code on a raspberry pi 2 (quad-core ARM7) 

you need a firmware 2.3.0

read my step by step tutorial for ubuntu x64
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg729604/#msg729604[/url

[tomas123]

@Bud

just for fun I changed the sample code from libuvc for 16 Bit RAW and appended the auto-level code from tmbinc
https://github.com/ktossell/libuvc/blob/master/src/test.c

compile the code like above described

Code: [Select]

# /usr/bin/cc  -o mytest mytest.c -luvc `pkg-config --cflags --libs opencv`
... some warnings

# sudo ./mytest
UVC initialized
Device found
Device opened
...

the source:

Code: [Select]

/*********************************************************************
* Software License Agreement (BSD License)
*
*  Copyright (C) 2010-2012 Ken Tossell
*  All rights reserved.
*
*  Redistribution and use in source and binary forms, with or without
*  modification, are permitted provided that the following conditions
*  are met:
*
*   * Redistributions of source code must retain the above copyright
*     notice, this list of conditions and the following disclaimer.
*   * Redistributions in binary form must reproduce the above
*     copyright notice, this list of conditions and the following
*     disclaimer in the documentation and/or other materials provided
*     with the distribution.
*   * Neither the name of the author nor other contributors may be
*     used to endorse or promote products derived from this software
*     without specific prior written permission.
*
*  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
*  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
*  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
*  FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
*  COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
*  INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
*  BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
*  LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
*  CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
*  LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
*  ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
*  POSSIBILITY OF SUCH DAMAGE.
*********************************************************************/
#include <stdio.h>
#include <opencv/highgui.h>

#include "libuvc/libuvc.h"

struct cb_context {
  FILE *out;
  struct timeval tv_start;
  int frames;
};

void cb(uvc_frame_t *frame, void *ptr) {
  uvc_frame_t *rgb;
  uvc_error_t ret;
  IplImage* cvImg;

  printf("callback! length = %u, ptr = %d\n", frame->data_bytes, (int) ptr);

  rgb = uvc_allocate_frame(frame->width * frame->height * 3);
  if (!rgb) {
    printf("unable to allocate rgb frame!");
    return;
  }
  ret = uvc_any2rgb(frame, rgb);
  if (ret) {
    uvc_perror(ret, "uvc_any2rgb");
    uvc_free_frame(rgb);
    return;
  }

  cvImg = cvCreateImageHeader(
      cvSize(rgb->width, rgb->height),
      IPL_DEPTH_8U,
      3);

  unsigned short *pix = frame->data;
  unsigned char *fb_proc;
  fb_proc = malloc(320 * 240);
  int i, x, y;
 
   // calc min, max, rms
  int min = 0x10000, max = 0;
  for (i = 0; i < 320 * 240; ++i) {
    if (pix[i] < min) min = pix[i];
    if (pix[i] > max) { max = pix[i]; }
  }
  int scale = 0x10000 / (max - min);

  // stretch level and shift 16 to 8 Bit 
  for (y = 0; y < 240; ++y)
  {
    for (x = 0; x < 320; ++x) {
      int v = (pix[y * 320 + x] - min) * scale >> 8;
      fb_proc[y * 320 + x] = v;
    }
  }
 
 
  CvSize image_size;
  image_size.height = 240;
  image_size.width = 320;
  int channels = 1;
  IplImage *image = cvCreateImageHeader(image_size, IPL_DEPTH_8U, channels);
  cvSetData(image, fb_proc, image->widthStep);
 
  cvNamedWindow("Test", CV_WINDOW_AUTOSIZE);
  cvShowImage("Test", image);
 
  cvWaitKey(10);

  cvReleaseImageHeader(&cvImg);

  uvc_free_frame(rgb);
}

int main(int argc, char **argv) {
  uvc_context_t *ctx;
  uvc_device_t *dev;
  uvc_device_handle_t *devh;
  uvc_stream_ctrl_t ctrl;
  uvc_error_t res;
  struct cb_context cb_ctx = {0};
  /* Initialize a UVC service context. Libuvc will set up its own libusb
   * context. Replace NULL with a libusb_context pointer to run libuvc
   * from an existing libusb context. */
  res = uvc_init(&ctx, NULL);

  if (res < 0) {
    uvc_perror(res, "uvc_init");
    return res;
  }

  puts("UVC initialized");

  res = uvc_find_device(
      ctx, &dev,
      0, 0, NULL);

  if (res < 0) {
    uvc_perror(res, "uvc_find_device");
  } else {
    puts("Device found");

    res = uvc_open(dev, &devh);

    if (res < 0) {
      uvc_perror(res, "uvc_open");
    } else {
      puts("Device opened");

      uvc_print_diag(devh, stderr);
    /* Try to negotiate a 640x480 30 fps YUYV stream profile */
      res = uvc_get_stream_ctrl_format_size(
          devh, &ctrl, UVC_FRAME_FORMAT_YUYV, 320, 240, 15  /* YUV 422, aka YUV 4:2:2. try _COMPRESSED */
      );
  /* Print out the result */
      uvc_print_stream_ctrl(&ctrl, stderr);

      if (res < 0) {
        uvc_perror(res, "get_mode");/* device doesn't provide a matching stream */
      } else {
       /* Start the video stream. The library will call user function cb:
         *   cb(frame, (void*) 12345)
         */
        res = uvc_start_streaming(devh, &ctrl, cb, &cb_ctx, 0);

        if (res < 0) {
          uvc_perror(res, "start_streaming"); /* unable to start stream */
        } else {
          puts("Streaming for 10 seconds...");
          uvc_error_t resAEMODE = uvc_set_ae_mode(devh, 1);
          uvc_perror(resAEMODE, "set_ae_mode");
          int i;
          for (i = 1; i <= 10; i++) {
            /* uvc_error_t resPT = uvc_set_pantilt_abs(devh, i * 20 * 3600, 0); */
            /* uvc_perror(resPT, "set_pt_abs"); */
            uvc_error_t resEXP = uvc_set_exposure_abs(devh, 20 + i * 5);
            uvc_perror(resEXP, "set_exp_abs");
           
            sleep(1);
          }
          sleep(20);
          uvc_stop_streaming(devh);
  puts("Done streaming.");
        }
      }

      uvc_close(devh);
      puts("Device closed");
    }

    uvc_unref_device(dev);
  }

  uvc_exit(ctx);
  puts("UVC exited");

  return 0;
}


RAW live stream:



@ tmbinc

What do you mean with this code snippset from fuchsjagd?
It's like a 20x20 pixel watermark.
You can see it in your video at the bottom on the left side.

Code: [Select]

  unsigned short *pix = frame->data;
  int x, y;
 
  for (y = 160; y < 180; ++y) {
    for (x = 60; x < 80; ++x) {
      pix[y * 320 + x] = pix[y * 320 + 60];
    }
  }
Thank you for the fuchsjagd code
I changed it for Ubuntu x64, only the frame buffer part it's a little bit tricky.

[Bud]

Ya baby, now we are talking! Thanks tomas123!

Though I had to update to Ubuntu 14.whatever, otherwise the OpenCV library did not compile. That is why I avoid Linux, the developers have no clue as to what flavor of Linux and what version their product works with. Guessing game every time and you better have all Developers packages in the Universe installed to mitigate the pain.

Anyway, your latest code worked and I took a liberty to tailor it a little. I made use of the original buffer initialization from the original test file, as there seemed to be some duplication in last version. I also commented out the frame data length message that was flooding the screen - I think we got the code working to the point we do not need to monitor that anymore. And a nice thing: I replaced CV_WINDOW_AUTOSIZE with CV_WINDOW_NORMAL, and now the window is resizable.



I also returned to waiting for a key press to end streaming vs a timeout.

BTW, why is this code only falls thru if ENTER key is pressed but not any other key ?

Code: [Select]

          // stream until key is pressed.
          char c;
          read(0, &c, 1);
          /* End the stream. Blocks until last callback is serviced */
          uvc_stop_streaming(devh);
          puts("Done streaming.");
 
So this actually is becoming a useful piece of software. You can make snapshots from the window (a sample attached). I think autoleveling may need to be improved, it did not always work well.

EDIT: attached the source

[Bud]

@tomas123

OK some more awkward code added to track mouse coordinates and display pixel value under the cursor using a slider control as a gauge meter



So this pretty much completes a Proof of Concept and now the actual pixel temperature can be calculated and displayed, you will find my comments in the code. I recall you were the one who reverse engineered FLIR temperature calculation? This at the moment is beyond my ability so pls see if you can update the code with it. Thks!

What else could be added is perhaps a center spot and box measures. I think this can be done with OpenCV.
For now you can position mouse cursor at the center of the image to emulate a center spot and then moving the camera will read temperature from it.

EDIT: just in case: I compile using the following command:

Code: [Select]

gcc -o flirfox2 flirfox2.c -luvc -lm `pkg-config --cflags --libs opencv`

[Bud]

@tomas123

And here is something interesting from me playing with window modes. When I used CV_WINDOW_OPENGL, moved the window on the screen, exploded to full screen then back, noticed something weird (selection is mine):



For some reason the camera changed frame data length and it seemed to stayed consistent until I returned from full screen mode. What the heck the camera was transmitting and how the heck the code managed to display a proper image??? 

Did not have time to play more with that, may be after adding streaming data to file will see if can replicate it and look what is in the extra data.

[tomas123]

I can confirm this additional bytes.
In my Ubuntu virtual machine I often have spontaneous glitches associated with changes of usb frame lenght.
With camera connected direct to host machine USB this glitches are rarely.
The additional bytes are without informations (most zeros)

I solved this glitches in the juchsjagd code by hardcoding the frame size with 320x240x2 Byte (crop the tail).

So this pretty much completes a Proof of Concept and now the actual pixel temperature can be calculated and displayed, you will find my comments in the code. I recall you were the one who reverse engineered FLIR temperature calculation? This at the moment is beyond my ability so pls see if you can update the code with it.

this is really hard
You can't grep a Flir temperature value from a 8 Bit grayscale video stream without additional informations (high/low value etc)
Therefore the best way is to enlarge the image size and imprint the temperature values (like max/min) outside the visible video.

the nice running fuchsjagd code from tmbinc contains all necessary code snippsets (font.h  )
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg736344/#msg736344

the mathematical equations for calculating RAW to temperature I posted here:
- sample with excel sheet: http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,4898.msg27546.html#msg27546
- sample with php code https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg342072/#msg342072

As attachment I post my excel sheet with all formulas and some unpublished  infos from Flir

The required Flir parameters are calibrated constants for every camera. You get this values from every radiometric jpg with this exiftool command:

Code: [Select]

>exiftool -Atmos* -Planck* -Object* -Emis* -Reflected* -Relativ* sample.jpg
Atmospheric Temperature         : 20.0 C
Atmospheric Trans Alpha 1       : 0.006569
Atmospheric Trans Alpha 2       : 0.012620
Atmospheric Trans Beta 1        : -0.002276
Atmospheric Trans Beta 2        : -0.006670
Atmospheric Trans X             : 1.900000
Planck R1                       : 15314.136
Planck B                        : 1404.6
Planck F                        : 1
Planck O                        : -978
Planck R2                       : 0.011565135
Object Distance                 : 1.00 m
Emissivity                      : 0.60
Reflected Apparent Temperature  : 20.0 C
Relative Humidity               : 50.0 %

[tmbinc]

What do you mean with this code snippset from fuchsjagd?
It's like a 20x20 pixel watermark.
You can see it in your video at the bottom on the left side.

Code: [Select]

  unsigned short *pix = frame->data;
  int x, y;
 
  for (y = 160; y < 180; ++y) {
    for (x = 60; x < 80; ++x) {
      pix[y * 320 + x] = pix[y * 320 + 60];
    }
  }

Haha, I did forget about that code. Sorry. I have some dirt on my lens (or sensor) which I didn't cleanup yet (out of fear to break something). This created a spot that read a slightly larger temperature, which triggered the "Fuchs temperature threshold" ("Ftt").

The, erm, solution was to just copy the area next to it over the spot.

[Pantheron]

Hello guys,

i tried this also, but i cant get a stream with the first flir.c release.
I also got some other streaming descriptors than you, probably you can help me out?

Thank you in advance.

It is an FLIR E4 wit FW 2.30 and hacked for Resolution and Menu (Firmware was original at 2.30)

Code: [Select]

root@iJudge:/home/pantheron/libuvc# build/flir e4x.raw
UVC initialized
Device found
unsupported descriptor subtype: 3
unsupported descriptor subtype: 13
unsupported descriptor subtype: 3
unsupported descriptor subtype: 13
Device opened
DEVICE CONFIGURATION (05ac:8510/CC2CAV0DA1DN9KE0) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 5
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 00
  size: 160x120
  bit rate: 24576000-147456000
  max frame size: 38400
  default interval: 1/29
  interval[0]: 1/29
  interval[1]: 1/25
  interval[2]: 1/23
  interval[3]: 1/14
FrameDescriptor(2)
  capabilities: 00
  size: 176x144
  bit rate: 24576000-147456000
  max frame size: 50688
  default interval: 1/29
  interval[0]: 1/29
  interval[1]: 1/25
  interval[2]: 1/23
  interval[3]: 1/14
FrameDescriptor(3)
  capabilities: 00
  size: 320x240
  bit rate: 24576000-147456000
  max frame size: 153600
  default interval: 1/29
  interval[0]: 1/29
  interval[1]: 1/25
  interval[2]: 1/23
  interval[3]: 1/14
FrameDescriptor(4)
  capabilities: 00
  size: 352x288
  bit rate: 24576000-147456000
  max frame size: 202752
  default interval: 1/29
  interval[0]: 1/29
  interval[1]: 1/25
  interval[2]: 1/23
  interval[3]: 1/14
FrameDescriptor(5)
  capabilities: 00
  size: 640x480
  bit rate: 24576000-147456000
  max frame size: 614400
  default interval: 1/29
  interval[0]: 1/29
  interval[1]: 1/25
  interval[2]: 1/23
  interval[3]: 1/14
FrameDescriptor(6)
  capabilities: 00
  size: 480x360
  bit rate: 24576000-147456000
  max frame size: 345600
  default interval: 1/29
  interval[0]: 1/29
  interval[1]: 1/25
  interval[2]: 1/23
  interval[3]: 1/14
FrameDescriptor(7)
  capabilities: 00
  size: 1280x720
  bit rate: 24576000-147456000
  max frame size: 1843200
  default interval: 1/10
  interval[0]: 1/10
MJPEGFormat(2)
  bits per pixel: 0
  GUID: 4d4a5047000000000000000000000000 (MJPG)
  default frame: 2
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 00
  size: 960x544
  bit rate: 24576000-147456000
  max frame size: 1044480
  default interval: 1/29
  interval[0]: 1/29
  interval[1]: 1/25
  interval[2]: 1/23
  interval[3]: 1/14
FrameDescriptor(2)
  capabilities: 00
  size: 1024x576
  bit rate: 24576000-147456000
  max frame size: 1179648
  default interval: 1/29
  interval[0]: 1/29
  interval[1]: 1/25
  interval[2]: 1/23
  interval[3]: 1/14
FrameDescriptor(3)
  capabilities: 00
  size: 1280x720
  bit rate: 24576000-147456000
  max frame size: 1843200
  default interval: 1/29
  interval[0]: 1/29
  interval[1]: 1/25
  interval[2]: 1/23
  interval[3]: 1/14
END DEVICE CONFIGURATION
bmHint: af90
bFormatIndex: 5
bFrameIndex: 157
dwFrameInterval: 32766
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 4176
dwMaxVideoFrameSize: 0
dwMaxPayloadTransferSize: 4197312
bInterfaceNumber: 254
get_mode: Invalid mode (-51)
Device closed
UVC exited
root@iJudge:/home/pantheron/libuvc#

[Pantheron]

Okay,

it tooks the isight camera of the macbook...
So i added the vendor and product id directly into the code for the flir e4 and now it works

[Bud]

Good point. Can you post the changed code line , thks.

[Bud]

Here is a version that streams data in a uncompressed 8-bit grayscale AVI file.  It can then be opened and colorized in ImageJ.

[tomas123]

Good point. Can you post the changed code line , thks.

Code: [Select]

  res = uvc_find_device(
      ctx, &dev,
      0, 0, NULL);
change 0,0 to vid,pid

https://int80k.com/libuvc/doc/group__device.html

uvc_error_t uvc_find_device   (   uvc_context_t *    ctx,
uvc_device_t **    dev,
int    vid,
int    pid,
const char *    sn
)      
Finds a camera identified by vendor, product and/or serial number.

Parameters
[in]   ctx   UVC context in which to search for the camera
[out]   dev   Reference to the camera, or NULL if not found
[in]   vid   Vendor ID number, optional
[in]   pid   Product ID number, optional
[in]   sn   Serial number or NULL

[Bud]

Thanks tomas123 for your input and information/files in your previous post!

[Bud]

Here is a version that supports both RAW 16-bit and grayscale 8-bit streaming to file

Usage:

./flirfox4               :Display video, no streaming  to file
./flirfox4 -v      :same as above
./flirfox4 -r  filename   :stream 16-bit RAW thermal video  to file, no header
./flirfox4 -g filename   :stream 8-bit grayscale auto-leveled video  to file with AVI format header

Both can be opened in ImageJ program, color table applied and can be saved in a variety of formats.

EDIT:   and it also has E4 PID/VID hardcoded, so it only be looking for E4 camera.
EDIT:   do not expect well written code from me. First, I am Linux impaired. Second, I heard about C language that there is such thing and that is about it. Third, I am not a programmer. 

[Bud]

Temperature calculation added. Try it folks if it works for you.
You will need a file called e4data.txt with your camera's calibration data. You create the file using exiftool as tomas123 explained here: https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg744673/#msg744673

You also need to make a sample image from your camera, exiftool extracts the data from it (google for exiftool download if you do not have it).

Run exiftool command against the sample E4 file:

Code: [Select]

exiftool -Atmos* -Planck* -Object* -Emis* -Reflected* -Relativ*  FLIR_sample.jpg >e4data.txt
Once made and inspected, place the e4data.txt file in the folder with the compiled executable. For now the program just displays temperature (at mouse pointer coordinates) in the top left corner. Compare it with readings on your camera by pointing the mouse and camera's crosshair to a same hot spot.

Thanks tomas123 for the formulas!

[tomas123]

  great work
unfortunately now I have not enough time to go deeper in your code

a fast check with Ubuntu results in a "nan" value for temperature (see screenshot)

Code: [Select]

$ exiftool -Atmos* -Planck* -Object* -Emis* -Reflected* -Relativ*  FLIR3344.jpg >e4data.txt

$ cat e4data.txt
Atmospheric Temperature         : 20.0 C
Planck R1                       : 13061.76
Planck B                        : 1352.2
Planck F                        : 2.7
Planck O                        : -7334
Planck R2                       : 0.027282318
Object Distance                 : 1.00 m
Emissivity                      : 0.95
Reflected Apparent Temperature  : 20.0 C
Relative Humidity               : 50.0 %

$ /usr/bin/cc  -o flirfox6 flirfox6.c -luvc `pkg-config --cflags --libs opencv` -lm
$ sudo ./flirfox6

I will tomorrow debug your nice code.


another note:

I compiled successful the libuvc patch on mac OS X 10.9.

my raw note for interested users (use macports):

Code: [Select]

//check
$ port installed | grep active
  cmake @3.3.1_0 (active)
  libusb @1.0.19_0 (active)
  jpeg @9a_1 (active)

$ git clone https://github.com/ktossell/libuvc
Cloning into 'libuvc'...
remote: Counting objects: 526, done.
remote: Total 526 (delta 0), reused 0 (delta 0), pack-reused 526
Receiving objects: 100% (526/526), 263.90 KiB | 309.00 KiB/s, done.
Resolving deltas: 100% (305/305), done.
Checking connectivity... done.

$ cd libuvc/

// now use  patch in the src folder
$ patch < libuvc.patch
   patching file stream.c

cd ..
$ mkdir build
$ cd build
$ cmake ..
-- The C compiler identification is AppleClang 5.1.0.5030040
-- The CXX compiler identification is AppleClang 5.1.0.5030040
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- No build type selected, default to Release
-- No target type selected, default to shared library
-- Found JPEG: /opt/local/lib/libjpeg.dylib 
-- Building libuvc with JPEG support.
-- Configuring done
CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   uvc

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/jfried/Documents/Flir/uvc/libuvc/build

$ make
Scanning dependencies of target uvc
[ 10%] Building C object CMakeFiles/uvc.dir/src/ctrl.c.o
[ 20%] Building C object CMakeFiles/uvc.dir/src/ctrl-gen.c.o
[ 30%] Building C object CMakeFiles/uvc.dir/src/device.c.o
[ 40%] Building C object CMakeFiles/uvc.dir/src/diag.c.o
[ 50%] Building C object CMakeFiles/uvc.dir/src/frame.c.o
[ 60%] Building C object CMakeFiles/uvc.dir/src/init.c.o
[ 70%] Building C object CMakeFiles/uvc.dir/src/stream.c.o
[ 80%] Building C object CMakeFiles/uvc.dir/src/misc.c.o
[ 90%] Building C object CMakeFiles/uvc.dir/src/frame-mjpeg.c.o
[100%] Linking C shared library libuvc.dylib
[100%] Built target uvc
the RAW streaming of Flir E4  works fine under OS X (sample code from tmbinc)

Code: [Select]

// fast check with my full pathes
$ cc -o flir flir.c -I/opt/local/include/ -I/Users/tomas/Documents/Flir/uvc/libuvc/build/include -I/Users/tomas/Documents/Flir/uvc/libuvc/include/  -L/Users/tomas/Documents/Flir/uvc/libuvc/build -luvc

$ ./flir
UVC initialized
usage: ./flir out.raw

./flir 1
UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x240
  bit rate: 912384-912384
  max frame size: 153600
  default interval: 1/3
  interval[0]: 1/15
  interval[1]: 1/7
  interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming...
got frame 1441305922:212093, 153600 bytes
got frame 1441305922:308068, 153600 bytes
got frame 1441305922:404825, 153600 bytes
got frame 1441305922:500120, 153600 bytes
got frame 1441305922:594571, 153600 bytes
...

now I have some trouble with OpenCV under OS X ...

[Bud]

@tomas123

Oh-oh, seems a problem with exiftool outputting less data when run in linux vs windows. I ran exiftool in Windows, it dumps 15 parameters in total. Yours seems to dump only 10, that is what caused the nan error.
I will check what version of exiftool executable i have in windows.

[Bud]

So my exiftool Windows version is 9.9, this is what it outputs:

Code: [Select]

Atmospheric Temperature         : 20.0 C
Atmospheric Trans Alpha 1       : 0.00656
Atmospheric Trans Alpha 2       : 0.01262
Atmospheric Trans Beta 1        : -0.0022
Atmospheric Trans Beta 2        : -0.0066
Atmospheric Trans X             : 1.90000
Planck R1                       : 14901.4
Planck B                        : 1400.9
Planck F                        : 2.5
Planck O                        : -7187
Planck R2                       : 0.02804
Object Distance                 : 1.00 m
Emissivity                      : 0.95
Reflected Apparent Temperature  : 20.0 C
Relative Humidity               : 50.0 %
@tomas123: seems yours does not output alpha/beta/X, not sure if this is how  exiftool works in Linux 

In the program I used the formulas with Distance from your Excel spreadsheet, that required I believe all 15 entries.
BTW, the last version is just using atmospheric temperature, object distance, reflected temperature and humidity read from the sample file. I intended later to replace them with sliders added to the video window so it would be possible to adjust for the current scene, since this information is not in the video stream.

[Bud]

Latest version,

• Temperature label moved to its own overlay so it will not appear in recorded files
• A crosshair cursor is added to read temperature, use the mouse to drag the cursor. Left mouse Double click resets position to the image center.
• The cursor appears in 8-bit grayscale video files, won't appear in 16-bit RAW video files.
• Added sliders to adjust a few ambient parameters .

For now I reverted to a fixed size window, as the sliders eat window space, shrinking the image. Would probably add a command line switch later to allow for window resize.

[tomas123]

great work
unfortunately now I have not enough time to go deeper in your code

sorry, that I'm wasted your time

with this steps on my virtual machine

Code: [Select]

$ lsb_release -a
Description: Ubuntu 14.04.1 LTS

$ sudo apt-get install exiftool
Note, selecting 'libimage-exiftool-perl' instead of 'exiftool'

$ sudo apt-get install libimage-exiftool-perl

$ exiftool -ver
9.46
I got an old exiftool version from Jan. 11, 2014: "Version 9.46 production release" (without the values for atmosphere transmissivity)
... and to make matters worse, I'm a coder of the Flir part of exiftool

I downloaded the current version 10.0.1 from Phils homepage and now all works fine 


If you prefer to parse XML in C than you can use the X parameter in exiftool:

Code: [Select]

Image-ExifTool-10.01/exiftool -X -Atmos* -Planck* -Object* -Emis* -Reflected* -Relativ* FLIR3344.jpg
<?xml version='1.0' encoding='UTF-8'?>
<rdf:RDF xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#'>
<rdf:Description rdf:about='FLIR3344.jpg'
  xmlns:et='http://ns.exiftool.ca/1.0/' et:toolkit='Image::ExifTool 10.01'
  xmlns:FLIR='http://ns.exiftool.ca/APP1/FLIR/1.0/'>
 <FLIR:AtmosphericTemperature>20.0 C</FLIR:AtmosphericTemperature>
 <FLIR:AtmosphericTransAlpha1>0.006569</FLIR:AtmosphericTransAlpha1>
 <FLIR:AtmosphericTransAlpha2>0.012620</FLIR:AtmosphericTransAlpha2>
 <FLIR:AtmosphericTransBeta1>-0.002276</FLIR:AtmosphericTransBeta1>
 <FLIR:AtmosphericTransBeta2>-0.006670</FLIR:AtmosphericTransBeta2>
 <FLIR:AtmosphericTransX>1.900000</FLIR:AtmosphericTransX>
 <FLIR:PlanckR1>18453.355</FLIR:PlanckR1>
 <FLIR:PlanckB>1460.6</FLIR:PlanckB>
 <FLIR:PlanckF>1</FLIR:PlanckF>
 <FLIR:PlanckO>-7003</FLIR:PlanckO>
 <FLIR:PlanckR2>0.013993904</FLIR:PlanckR2>
 <FLIR:ObjectDistance>1.00 m</FLIR:ObjectDistance>
 <FLIR:Emissivity>0.95</FLIR:Emissivity>
 <FLIR:Emissivity>0.95</FLIR:Emissivity>
 <FLIR:ReflectedApparentTemperature>20.0 C</FLIR:ReflectedApparentTemperature>
 <FLIR:RelativeHumidity>50.0 %</FLIR:RelativeHumidity>
</rdf:Description>
</rdf:RDF>

[tomas123]

Because there are some minor differences between the temperature of your crosshair cursor and the E4 screen I rechecked my excel sheet with a sample image.

(1) load image in Flir Tools and export the temperature off all pixels in a csv file
(2) with exiftool and imagemagick export the raw values of some pixels
(3) with Excel compare calculated temperature with exported values from Flir Tools

The values in Flir Tools and my Excel sheet are identical with all digits (milliKelvin) 

The calculations in your code are also right. I checked it with:

Code: [Select]

185: sprintf(szText, "%3.1f C / RAW: %i",t, pix[cpY * 320 + cpX]);The temperature differences under crosshair I can't explain with averaging of 2x2 or 3x3 pixels on the E4 display 

all steps in bash

Code: [Select]

// export with Flir Tools the temperature of all Pixels (Flir Tools export as *.csv)
$ head -n11 FLIR0080.csv
File: ;C:\Users\admin\Desktop\1\FLIR0080.jpg

Parameters:;Emissivity:;0,95
;Refl. temp.:;20 °C
;Distance:;1 m
;Atmospheric temp.:;20 °C
;Ext. optics temp.:;20 °C
;Ext. optics trans.:;1
;Relative humidity:;50 %
;
Frame 1;18,359;18,373;18,373;18,359;18,387;18,429;18,401;18,457;18,457;18,443;18,443;18,429;18,415;18,415;18,401;18,401;18,401;18,387;18,387;18,373;18,359;18,345;18,331

// print a readable form
s jfried$ grep Frame FLIR0080.csv | tr ";" "\n" | head -n 17
Frame 1
18,359
18,373
18,373
18,359
18,387
18,429
18,401
18,457
18,457
18,443
18,443
18,429
18,415
18,415
18,401
18,401

// get Flir parameters for excel sheet
$ /Image-ExifTool-10.01/exiftool -Atmos* -Planck* -Object* -Emis* -Reflected* -Relativ* FLIR0080.jpg
Atmospheric Temperature         : 20.0 C
Atmospheric Trans Alpha 1       : 0.006569
Atmospheric Trans Alpha 2       : 0.012620
Atmospheric Trans Beta 1        : -0.002276
Atmospheric Trans Beta 2        : -0.006670
Atmospheric Trans X             : 1.900000
Planck R1                       : 15119.786
Planck B                        : 1405.2
Planck F                        : 2.5
Planck O                        : -7652
Planck R2                       : 0.027600825
Object Distance                 : 1.00 m
Emissivity                      : 0.95
Reflected Apparent Temperature  : 20.0 C
Relative Humidity               : 50.0 %

// export RAW values of radiometric image
$ exiftool -b -RawThermalImage FLIR0080.jpg > FLIR0080.png

// swap bytes
$ convert -define png:swap-bytes=on FLIR0080.png FLIR0080swap.png

// print RAW values of first 16 Pixel (top left)
$ for i in $(seq 0 15); do convert FLIR0080swap.png -crop 1x1+$i+0 -format "%[fx:round(65535*u.r)]" info:;echo ; done
12167
12168
12168
12167
12169
12172
12170
12174
12174
12173
12173
12172
12171
12171
12170
12170
a pixel around the center of image (the crosshair) 
$ convert FLIR0080swap.png -crop 1x1+160+120 -format "%[fx:round(65535*u.r)]" info:
12644
=> 24,785 °C (see imprinted spot 24,8°C)

see my links in signature for details
and https://www.eevblog.com/forum/testgear/flir-e4-thermal-imaging-camera-teardown/msg350250/#msg350250

[Bud]

Thanks tomas123. Could the difference be because of rounding errors in Linux math, the formulas have power of 2 and 3 as well as square roots, so rounding errors may accumulate. I think what can be done is to run the temp calculation routine with a known input value and check results of each step in the routine against your spreadsheet. This way we can catch where the difference occurs.

[Bud]

@tomas123
That is interesting the doctored libuvc works on Mac OS. Nice. I guess may also work on Win10, i heard it overcomes the limitations that prevented libuvc from working with camera streaming in Win7.

[tomas123]

The differences are only visible in rare situations at higher temperatures and only small (<0,5K).
I'm not sure, but it doesn't look like a average of 4x4 pixels.
Also this differences are not rounding errors.

It's possible, that the E4 raw video stream contains less "magic Flir" post procession filters like the radiometric images (jpg).

[Bud]

Must be some secret sauce in E4, since temperature calculation in  the program matches pretty well your Excel spreadsheet, I checked for a pixel level 10000:

T_Obj   -8.397801     -8.397789

Anyway, attached is the version with  MJPG compression added ( -m switch, will save 8-bit MJPG compressed file).
I will probably stop here for now.

EDIT: I think this program with some additions may be useful as a data logger for rapidly changing temperatures, since readings are available for each frame. Even this version can be used for that. Write video to a 8-bit file, open the file in any video viewer and step through the frames to read temperature at the cursor.

[tomas123]

I will probably stop here for now.

You've done a great job 
I like your solution for temperature measurement under the free movable crosshair.

I did not have enough time to work with my raspberry pi (spycam for  animals).
The OpenVC window is not the best solution for a ssh remote session disconnected with screen.
As a next step, I'll try to send the 8 Bit video stream to a v4l2loopback device (a video for linux loopback device like /dev/video1).

PS: I wasted many time for running your c code under Mac OS X.
The OpenCV library under OS X is a pain. I have unresolved problems with the OpenCV heap inside the function cb(frame,ptr) and the Qt library.
I think the best solution is switching to the newer opencv c++ api
But I give up 

As I wrote here, the patched libuvc library works stable under OS X:
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg747569/#msg747569

[Bud]

Yes I did not exactly like OpenCV but was easier for an inexperienced person like me to come up with some code. I run Ubuntu 14 on a Win7 guest using Oracle VirtualBox (32Gb RAM) and OpenCV appears to be glitchy, not sure if this is because it runs on a VM. It also seems to write 9 fps video with a 15fps header which causing the video to play too fast. Not sure how it ends up with 9 fps.

Anyway, this is what I call "better than nothing" and I tried to summarize the steps below:

The purpose of this topic and this program was to get FLIR E4 camera to stream RAW thermal 16-bit video that can be written in a file as uncompressed 16-bit grayscale video for further processing elsewhere. As a by side feature, the program can display a rescaled 8-bit video in a window and can also write 8-bit grayscale uncompressed and MJPG compressed files that include temperature measurement under a spot cursor that can be moved/set in the program using the mouse. The program also have a few adjustment controls that allow to change several environmental parameters and obtain corrected temperature measurements in real time.

The program seems to only work in Linux at this time because the underlying libraries (libuvc and OpenCV) seem to not provide required functionality in Win7 or MacOS.

Steps to get it running in Linux (I used Ubuntu 14):

• Download libuvc source and patch it as in this post from tomas123:
  https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg729604/#msg729604
  (use the patch file provided by user tmbinc in the zip archive attached to the very first message in this topic)
• Download and Compile OpenCV library as in this post from tomas123 (step 2)
  https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg742931/#msg742931
  
• Compile the program code
     gcc -o flirfox9 flirfox9.c -luvc -lm `pkg-config --cflags --libs opencv`
• Take a sample image from your camera, use exiftool to extract calibration constants for your camera from it (google for exiftool download if you do not have it).
  Run exiftool command against the sample E4 file:
     exiftool -Atmos* -Planck* -Object* -Emis* -Reflected* -Relativ*  FLIR_sample.jpg >e4data.txt
  Place the produced e4data.txt file in the folder with the compiled executable.
• Connect the camera and run the program
  ./flirfox9  [options]

Options:
 -r   filename  :write 16-bit RAW data to file
 -g  filename  :write 8-bit grayscale data to file
 -m filename  :write 8-bit grayscale MJPG compressed data to file
 -v                :only display 8-bit grayscale video

Use the mouse to click-n-drug the temperature cursor to the desired position. left mouse dbl-click resets the cursor to the image center.

For post processing you may want to take a look at a nice piece of software called ImageJ
http://imagej.nih.gov/ij/

It has bunch of image processing options plus bunch of color palettes (LUTs) to color the video and resave in various formats.

[tomas123]

a great tutorial

I edited my entry post on the first page and added a link list for new readers:
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg729604/#msg729604

[tomas123]

some results from playing around with the code from tmbinc  :

hardware: a raspberry pi 2 with a 5 inch hdmi display from waveshare
http://www.waveshare.com/wiki/5inch_HDMI_LCD

raspberry image: 2015-05-05-raspbian-wheezy.img

first install patched uvclib to get the raw video stream from a Flir Ex
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg729604/#msg729604
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg623251/#msg623251

I changed the source from tmbinc for a 32 bit framebuffer...
https://www.eevblog.com/forum/testgear/flir-ex-realtime-raw-radiometric-data-streaming-via-uvc/msg736344/#msg736344
...for my display with 800x480 in 16 Bit depth (RGB565) and added some comments for using the framebuffer.

For better viewing I rescaled the video to 640x480.

source: flirfox2x.c
compile with

Code: [Select]

/usr/bin/cc  -o flirfox2x ./flirfox2x.c -luvc -lm
using

Code: [Select]

$ sudo ./flirfox2x out.raw 20000
UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x240
  bit rate: 912384-912384
  max frame size: 153600
  default interval: 1/3
  interval[0]: 1/15
  interval[1]: 1/7
  interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
Streaming...
[0b397000] got frame 1442439612:475769, 153600 bytes, [12419..14340 rms 12865.158203 rmax 143]
[0b397000] got frame 1442439612:564221, 153600 bytes, [12415..14340 rms 12864.794922 rmax 284]
[0b397000] got frame 1442439612:666238, 153600 bytes, [12416..14339 rms 12864.424805 rmax 424]
[0b397000] got frame 1442439612:764188, 153600 bytes, [12417..14342 rms 12864.861328 rmax 563]
[0b397000] got frame 1442439612:866825, 153600 bytes, [12417..14339 rms 12864.305664 rmax 700]
you can clean your screen from frame buffer output with

Code: [Select]

xrefresh -d :0.0


next step is writing the video to a V4L2 loopback devices ( /dev/video0 )
https://github.com/umlaeute/v4l2loopback

The installation of v4l2loopback on a raspberry 2 with  2015-05-05-raspbian-wheezy.img was tricky, because there are no sources included for building kernel modules.
(no  /lib/modules/$(uname -r)/build )
All downloads to get the kernel sources for  3.18.7-v7+ fails 

I found here a working solution with roll back from  3.18.7-v7+ to the older kernel 3.18.0
https://github.com/umlaeute/v4l2loopback/issues/84

Code: [Select]

sudo aptitude install -y linux-image-rpi2-rpfv linux-headers-rpi2-rpfv
echo -e "kernel=vmlinuz-3.18.0-trunk-rpi2\ninitramfs=initrd.img-3.18.0-trunk-rpi followkernel" | sudo tee -a /boot/config.txt
sudo reboot
the next steps for installing v4l2loopback are simple

Code: [Select]

$ uname -a
Linux raspberryLCD 3.18.0-trunk-rpi2 #1 SMP PREEMPT Debian 3.18.5-1~exp1+rpi19 (2015-08-08) armv7l GNU/Linux

$ git clone [url]https://github.com/umlaeute/v4l2loopback[/url]

$ cd v4l2loopback

$ make
Building v4l2-loopback driver...
make -C /lib/modules/`uname -r`/build M=/home/pi/v4l2loopback/v4l2loopback modules
make[1]: Entering directory '/usr/src/linux-headers-3.18.0-trunk-rpi2'
Makefile:10: *** mixed implicit and normal rules: deprecated syntax
  CC [M]  /home/pi/v4l2loopback/v4l2loopback/v4l2loopback.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/pi/v4l2loopback/v4l2loopback/v4l2loopback.mod.o
  LD [M]  /home/pi/v4l2loopback/v4l2loopback/v4l2loopback.ko
make[1]: Leaving directory '/usr/src/linux-headers-3.18.0-trunk-rpi2'

$ sudo su

# make install
make -C /lib/modules/`uname -r`/build M=/home/pi/v4l2loopback/v4l2loopback modules_install
make[1]: Entering directory '/usr/src/linux-headers-3.18.0-trunk-rpi2'
Makefile:10: *** mixed implicit and normal rules: deprecated syntax
  INSTALL /home/pi/v4l2loopback/v4l2loopback/v4l2loopback.ko
  DEPMOD  3.18.0-trunk-rpi2
make[1]: Leaving directory '/usr/src/linux-headers-3.18.0-trunk-rpi2'
depmod -a  `uname -r`

$ sudo modprobe v4l2loopback

$ lsmod
Module                  Size  Used by
v4l2loopback           23372  0

...
source: flirfox-loop.c
compile

Code: [Select]

/usr/bin/cc  -o flirfox-loop ./flirfox-loop.c -luvc -lm
I simple mixed tmbinc source with the v4l2loopback test.c file 
source: https://github.com/umlaeute/v4l2loopback/blob/master/examples/test.c

using

Code: [Select]

$ sudo ./flirfox-loop out.raw 20000
UVC initialized
Device found
Device opened
DEVICE CONFIGURATION (09cb:1007/[none]) ---
Status: idle
VideoControl:
bcdUVC: 0x0100
VideoStreaming(1):
bEndpointAddress: 130
Formats:
UncompressedFormat(1)
  bits per pixel: 16
  GUID: 5955593200001000800000aa00389b71 (YUY2)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x240
  bit rate: 912384-912384
  max frame size: 153600
  default interval: 1/3
  interval[0]: 1/15
  interval[1]: 1/7
  interval[2]: 1/3
END DEVICE CONFIGURATION
bmHint: 0000
bFormatIndex: 1
bFrameIndex: 1
dwFrameInterval: 666666
wKeyFrameRate: 0
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 1
dwMaxVideoFrameSize: 153600
dwMaxPayloadTransferSize: 157696
bInterfaceNumber: 1
using output device: /dev/video0
     vid_format->type                =2
     vid_format->fmt.pix.width       =320
     vid_format->fmt.pix.height      =240
     vid_format->fmt.pix.pixelformat =1497715271
     vid_format->fmt.pix.sizeimage   =76800
     vid_format->fmt.pix.field       =1
     vid_format->fmt.pix.bytesperline=320
     vid_format->fmt.pix.colorspace  =8
Streaming...
[0b34c000] got frame 1442439057:790436, 153600 bytes, [12283..12491 rms 12423.203125 rmax 124]
[0b34c000] got frame 1442439057:887940, 153600 bytes, [12284..12485 rms 12422.550781 rmax 247]
[0b34c000] got frame 1442439057:988181, 153600 bytes, [12280..12489 rms 12420.328125 rmax 369]
[0b34c000] got frame 1442439058:88068, 153600 bytes, [12279..12486 rms 12418.430664 rmax 490]
[0b34c000] got frame 1442439058:187958, 153600 bytes, [12281..12484 rms 12418.650391 rmax 609]
[0b34c000] got frame 1442439058:297959, 153600 bytes, [12281..12481 rms 12419.558594 rmax 727]

You can start flirfox-loop in the background (as sample a screen session) and log out.

The program writes the screen video to /dev/video0 device.
From this point you can use all video players or webcam programs to pass the video from /dev/video0 to a html page etc...

As sample screenshot from a ssh session with x11 forwarding the mplayer

Code: [Select]

$ ssh -X pi@192.168.1.70
$ mplayer tv:// -tv driver=v4l2:device=/dev/video0
MPlayer svn r34540 (Raspbian), built with gcc-4.6 (C) 2000-2012 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing tv://.
TV file format detected.
Selected driver: v4l2
 name: Video 4 Linux 2 input
 author: Martin Olschewski <olschewski@zpr.uni-koeln.de>
 comment: first try, more to come ;-)
v4l2: your device driver does not support VIDIOC_G_STD ioctl, VIDIOC_G_PARM was used instead.
Selected device: Dummy video device (0x0000)
 Capabilities:  video capture  read/write  streaming
 supported norms:
 inputs: 0 = loopback;
 Current input: 0
 Current format: GREY


[tomas123]

unfortunately I have some trouble with charging the Ex with a raspberry 2

the max_usb_current switch doesn't work:

Code: [Select]

$ sudo nano /boot/config.txt
safe_mode_gpio=4
max_usb_current=1 

the only solution for me was using a USB hub with a power supply 

[Bud]

Wow, very nice!
 

[tomas123]

try the V4L2 loopback devices under Ubuntu ( /dev/video0 )

it's simple

Code: [Select]

$ sudo apt-get install v4l2loopback-dkms
Setting up v4l2loopback-dkms (0.8.0-1) ...

$ sudo modprobe v4l2loopback

// check your video devices video0 / video1
$ ls /dev/video*

- edit flirfox-loop.c for your loopback device video0 / video1
- compile flirfox-loop.c with

Code: [Select]

/usr/bin/cc  -o flirfox-loop ./flirfox-loop.c -luvc -lm

read the v4l2loopback wiki for nice samples
https://github.com/umlaeute/v4l2loopback/wiki

• GStreamer
• FFmpeg
• MPlayer
• Skype

ffserver is a nice tool for a webcam server
google for:
    ffserver /dev/video0

[Bud]

unfortunately I have some trouble with charging the Ex with a raspberry 2

It is not always working even with a conventional computer. I think FLIR did a poor job with implementation of the E4 USB charging circuit.

[tomas123]

@admin

Please move this thread with essential informations about Flir E4 RAW streaming to the new subforum "Thermal Imaging"

Thank You

[eg14]

Can someone please help me? I'm on Mac. Trying to stream raw data from my E30bx. the frame size is 153600 and I'm just getting a stream of what's on the display, not the raw video. Any ideas?

[Bud]

There is a setting in the config file to enable radiometric video. If you know how to do that you can check if it is present and add it if it is not there. Not sure if it also can be set  manually using rset command in Telnet, but you can try.

[eg14]

I do have that enabled. I noticed in Tomas's log from his E40 he had this

Code: [Select]

Streaming...
got frame 1439394751:346415, 157440 bytes
got frame 1439394751:411283, 157440 bytes
got frame 1439394751:479331, 157440 bytes

whereas I have this

Code: [Select]

Streaming...
got frame 1509121947:109894, 153600 bytes
got frame 1509121947:176200, 153600 bytes
got frame 1509121947:241827, 153600 bytes
got frame 1509121947:307965, 153600 bytes

The frames are too small to be radiometric.

When it's setting up the stream I see that the radiometric option is there...

Code: [Select]

UncompressedFormat(2)
  bits per pixel: 16
  GUID: 46374d3000001000800000aa00389b71 (F7M0)
  default frame: 1
  aspect ratio: 0x0
  interlace flags: 00
  copy protect: 00
FrameDescriptor(1)
  capabilities: 03
  size: 320x246
  bit rate: 912384-912384
  max frame size: 157440
  default interval: 1/7
  interval[0]: 1/30
  interval[1]: 1/15
  interval[2]: 1/7

but it doesn't seem to grab that stream.

[eg14]

Making some progress. I had to checkout version 6ff39f17e3f3a26f437bc6b3c09cc6c289db7891 of libuvc. Now my frames are the correct size, but when I split them neither Imagemagick nor exiftool recognize them as being images.

[eg14]

More progress. I used Tomas's split.pl script. I had to modify the magic bytes line to match mine

Code: [Select]

#!/usr/bin/perl
undef $/;
$_ = <>;
$n = 100;
# FLIR camera E40         
$pat="\x46\x46\x46\x00\x55\x56";
for $content (split(/(?=$pat)/)) {
        open(OUT, ">seq" . ++$n . ".fff");
        binmode OUT;
        print OUT $content;
        close(OUT);
}

From my raw data file:

Code: [Select]

00025800  46 46 46 00 55 56 43 00  00 00 00 00 00 00 00 00  |FFF.UVC.........|
00025810  00 00 00 00 64 00 00 00  40 00 00 00 02 00 00 00  |....d...@.......|
00025820  03 00 00 00 01 00 00 00  00 00 00 00 00 00 00 00  |................|
00025830  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00025840  20 00 00 00 71 00 00 00  01 00 00 00 80 00 00 00  | ...q...........|
now exiftool "sort of" recognizes that the frames are thermal images

Code: [Select]

ExifTool Version Number         : 10.55
File Name                       : seq102.fff
Directory                       : .
File Size                       : 154 kB
File Modification Date/Time     : 2017:10:27 13:32:26-05:00
File Access Date/Time           : 2017:10:27 13:40:12-05:00
File Inode Change Date/Time     : 2017:10:27 13:32:26-05:00
File Permissions                : rw-r--r--
File Type                       : FLIR
File Type Extension             : fff
MIME Type                       : application/unknown
Creator Software                : UVC
Emissivity                      : 0.95
Object Distance                 : 1.00 m
Reflected Apparent Temperature  : 20.0 C
Atmospheric Temperature         : 20.0 C
IR Window Temperature           : 20.0 C
IR Window Transmission          : 1.00
Relative Humidity               : 50.0 %
Planck R1                       : 14659.802
Planck B                        : 1391.5
Planck F                        : 1
Atmospheric Trans Alpha 1       : 0.006569
Atmospheric Trans Alpha 2       : 0.012620
Atmospheric Trans Beta 1        : -0.002276
Atmospheric Trans Beta 2        : -0.006670
Atmospheric Trans X             : 1.900000
Camera Temperature Range Max    : 120.0 C
Camera Temperature Range Min    : -20.0 C
Camera Temperature Max Clip     : 670.0 C
Camera Temperature Min Clip     : -40.0 C
Camera Temperature Max Warn     : 720.0 C
Camera Temperature Min Warn     : -20.0 C
Camera Temperature Max Saturated: 150.0 C
Camera Temperature Min Saturated: -60.0 C
Camera Model                    : FLIR E30bx
Camera Part Number              : 49001-1901
Camera Serial Number            : 49023364
Camera Software                 : 20.0.0
Lens Model                      : FOL18
Lens Part Number                :
Lens Serial Number              :
Field Of View                   : 25.0 deg
Filter Model                    :
Filter Part Number              :
Filter Serial Number            :
Planck O                        : -6278
Planck R2                       : 0.011133576
Raw Value Median                : 18122
Raw Value Range                 : 1124
Date/Time Original              : 2017:10:27 13:07:46.241-06:00
Focus Step Count                : 1522
Focus Distance                  : 17.2 m
Raw Thermal Image Width         : 17920
Raw Thermal Image Height        : 21846
Warning                         : Unrecognized FLIR RawThermalImage data format
Raw Thermal Image Type          : DAT
Raw Thermal Image               : (Binary data 153568 bytes, use -b option to extract)
Peak Spectral Sensitivity       : 10.3 um

this part is obviously an issue

Code: [Select]

Raw Thermal Image Width         : 17920
Raw Thermal Image Height        : 21846
Warning                         : Unrecognized FLIR RawThermalImage data format
Raw Thermal Image Type          : DAT
Raw Thermal Image               : (Binary data 153568 bytes, use -b option to extract)
Peak Spectral Sensitivity       : 10.3 um

[eg14]

Closer.... need to figure out a way to deal with the 3840 bytes of crap at the end of each frame

[cq-317]

I read this post, but there is still a little understanding. Can someone help me?

1.When connecting e4 wifi via usb, flir tools can choose whether to signal or video, and flir tools+ can record video directly. Does the method of modifying libuvc not apply to e4 wifi?

2.After getting the raw data stream through uvc, how can I convert it into Seq file?

[tmbinc]

I don't have a Flir E4 Wifi. Can the Flir E4 Wifi record 14-bit raw data, as opposed to remapped (colorized) video as visible on the screen?

[cq-317]

Yes, if you connect e4 wifi via usb, you can choose to display 14-bit raw data or video as visible on the screen in flir tools（ver 5.0 or higher）, which looks like the same as E40 (I don't have E40), but if you want to record it as seq file, You can only use flir tools+. The seq files recorded with flir tools+ can be opened in flir tools and edited. Flir tools+ is a paid software that can be used for free for one month.

[Bud]

But this topic does not have to do with flirtools. Not sure why you are asking this question here.

[cq-317]

I mean that flir tools can read the 14-bit raw data from E4 WIFI. Does the method of patch libuvc have been invalidated? Can we use the driver of flir directly to get the 14-bit raw data in e4 wifi without patch for libuvc? .

[Bud]

Sorry i still do not understand why you bringing flir tools to this thread. The work done in this thread did not involve flir tools. Perhaps you are asking a generic question "if flir tools can do this or that", in which case you better open a separate topic in Thermal Imaging subforum.

[tmbinc]

I think it's a valid question - the original "idea" of this thread was that Flir tools didn't offer to capture raw data, but yet we found a method to do so.

If FLIR tools can now (I assume with a newer device firmware) do this, then yes, this deprecates this "hack". I would be curious how the FLIR tools to it - maybe they fixed the UVC descriptor?

[Bud]

I see. I did not try fiir tools with E4 wifi. Still, this solution in this topic is much more portable, if to run on a Pi.

[Uno winner]

Does it work with flir i5 cam ?

[asmart]

For other model with radiometric IR video streaming over WiFi (such as T640), is this approach also works for RTSP connection? e.g contained raw data in each frame?

[Spirit532]

RTSP is a compressed MJPEG stream, so no, it wouldn't work.
You need USB.