Author Topic: CSQ file format  (Read 19930 times)

0 Members and 1 Guest are viewing this topic.

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
CSQ file format
« on: October 23, 2017, 01:58:21 am »
I was wondering if anyone has experience with extracting radiometric information from FLIR's csq file format (CSQ=compressed sequence file)?

I am looking to import the data into ImageJ for further analysis, and using FLIR tools or ResearchIR to convert the file format is slow when you have thousands of video files. 

Here is a sample csq in case this file type is not familiar. 

https://www.dropbox.com/s/ec6i18egpzkv7e6/IR_2017-05-19_0222.csq?dl=0

Cheers
Glenn
 

Offline tomas123

  • Frequent Contributor
  • **
  • Posts: 832
  • Country: de
Re: CSQ file format
« Reply #1 on: October 24, 2017, 07:56:39 pm »
looks like a regular flir sequence file
Code: [Select]
$ exiftool IR_2017-05-19_0222.csq
ExifTool Version Number         : 10.40
File Name                       : IR_2017-05-19_0222.csq
Directory                       : .
File Size                       : 27 MB
File Modification Date/Time     : 2017:10:24 21:52:16+02:00
File Access Date/Time           : 2017:10:24 21:52:11+02:00
File Inode Change Date/Time     : 2017:10:24 21:52:16+02:00
File Permissions                : rw-r--r--
File Type                       : FLIR
File Type Extension             : fff
MIME Type                       : application/unknown
Creator Software                : RTP
Emissivity                      : 0.95
Object Distance                 : 1.00 m
Reflected Apparent Temperature  : 20.0 C
Atmospheric Temperature         : 20.0 C
IR Window Temperature           : 20.0 C
IR Window Transmission          : 1.00
Relative Humidity               : 50.0 %
Planck R1                       : 21546.203
Planck B                        : 1507.2
Planck F                        : 1
Atmospheric Trans Alpha 1       : 0.006569
Atmospheric Trans Alpha 2       : 0.012620
Atmospheric Trans Beta 1        : -0.002276
Atmospheric Trans Beta 2        : -0.006670
Atmospheric Trans X             : 1.900000
Camera Temperature Range Max    : 150.0 C
Camera Temperature Range Min    : -40.0 C
Camera Temperature Max Clip     : 160.0 C
Camera Temperature Min Clip     : -60.0 C
Camera Temperature Max Warn     : 150.0 C
Camera Temperature Min Warn     : -40.0 C
Camera Temperature Max Saturated: 180.0 C
Camera Temperature Min Saturated: -60.0 C
Camera Model                    : FLIR T1030sc 12__
Camera Part Number              : 72501-0201
Camera Serial Number            : 72501090
Camera Software                 : 33.0.0
Lens Model                      : FOL83
Lens Part Number                : T198942
Lens Serial Number              : 74200341
Field Of View                   : 11.9 deg
Filter Model                    :
Filter Part Number              :
Filter Serial Number            :
Planck O                        : -6331
Planck R2                       : 0.016229488
Raw Value Median                : 14976
Raw Value Range                 : 2384
Date/Time Original              : 2017:05:19 12:45:33.583-07:00
Focus Step Count                : 65535
Focus Distance                  : 0.7 m
Raw Thermal Image Width         : 1024
Raw Thermal Image Height        : 768
Raw Thermal Image Type          : JPG
Raw Thermal Image               : (Binary data 209360 bytes, use -b option to extract)
Meas 1 Type                     : Spot
Meas 1 Params                   : 512 384
Meas 1 Label                    : 1
Palette Colors                  : 224
Above Color                     : 170 128 128
Below Color                     : 50 128 128
Overflow Color                  : 67 216 98
Underflow Color                 : 41 110 240
Isotherm 1 Color                : 100 128 128
Isotherm 2 Color                : 100 110 240
Palette Method                  : 0
Palette Stretch                 : 2
Palette File Name               : /FLIR/usr/etc/iron.pal
Palette Name                    : Iron
Palette                         : (Binary data 672 bytes, use -b option to extract)
GPS Map Datum                   : WGS84
Peak Spectral Sensitivity       : 9.5 um


look at your magic byte sequence and split the video in single frames
http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=5279.msg25567#msg25567

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #2 on: October 24, 2017, 09:09:29 pm »
It's the binary raw data I was hoping to share.  It is not in a format I recognise and wondered if you could take a look if you have time. 

Thank you!
 

Offline tomas123

  • Frequent Contributor
  • **
  • Posts: 832
  • Country: de
Re: CSQ file format
« Reply #3 on: October 24, 2017, 09:58:15 pm »
now I understand your question  ;)

Code: [Select]
$ cat split.pl
#!/usr/bin/perl
undef $/;
$_ = <>;
$n = 100;
# FLIR camera E40         
$pat="\x46\x46\x46\x00\x52\x54";
# Flir Tools (comment out)
#$pat = "\x46\x46\x46\x00\x43\x41\x50";
for $content (split(/(?=$pat)/)) {
        open(OUT, ">seq" . ++$n . ".fff");
        binmode OUT;
        print OUT $content;
        close(OUT);
}

now split your video
Code: [Select]
$ perl -f split.pl IR_2017-05-19_0222.csq
we got 167 frames
Code: [Select]
$ ls *.fff
seq101.fff seq117.fff seq133.fff seq149.fff seq165.fff seq181.fff seq197.fff seq213.fff seq229.fff seq245.fff seq261.fff
seq102.fff seq118.fff seq134.fff seq150.fff seq166.fff seq182.fff seq198.fff seq214.fff seq230.fff seq246.fff seq262.fff
seq103.fff seq119.fff seq135.fff seq151.fff seq167.fff seq183.fff seq199.fff seq215.fff seq231.fff seq247.fff seq263.fff
seq104.fff seq120.fff seq136.fff seq152.fff seq168.fff seq184.fff seq200.fff seq216.fff seq232.fff seq248.fff seq264.fff
seq105.fff seq121.fff seq137.fff seq153.fff seq169.fff seq185.fff seq201.fff seq217.fff seq233.fff seq249.fff seq265.fff
seq106.fff seq122.fff seq138.fff seq154.fff seq170.fff seq186.fff seq202.fff seq218.fff seq234.fff seq250.fff seq266.fff
seq107.fff seq123.fff seq139.fff seq155.fff seq171.fff seq187.fff seq203.fff seq219.fff seq235.fff seq251.fff seq267.fff
seq108.fff seq124.fff seq140.fff seq156.fff seq172.fff seq188.fff seq204.fff seq220.fff seq236.fff seq252.fff seq268.fff
seq109.fff seq125.fff seq141.fff seq157.fff seq173.fff seq189.fff seq205.fff seq221.fff seq237.fff seq253.fff
seq110.fff seq126.fff seq142.fff seq158.fff seq174.fff seq190.fff seq206.fff seq222.fff seq238.fff seq254.fff
seq111.fff seq127.fff seq143.fff seq159.fff seq175.fff seq191.fff seq207.fff seq223.fff seq239.fff seq255.fff
seq112.fff seq128.fff seq144.fff seq160.fff seq176.fff seq192.fff seq208.fff seq224.fff seq240.fff seq256.fff
seq113.fff seq129.fff seq145.fff seq161.fff seq177.fff seq193.fff seq209.fff seq225.fff seq241.fff seq257.fff
seq114.fff seq130.fff seq146.fff seq162.fff seq178.fff seq194.fff seq210.fff seq226.fff seq242.fff seq258.fff
seq115.fff seq131.fff seq147.fff seq163.fff seq179.fff seq195.fff seq211.fff seq227.fff seq243.fff seq259.fff
seq116.fff seq132.fff seq148.fff seq164.fff seq180.fff seq196.fff seq212.fff seq228.fff seq244.fff seq260.fff

see your frame rate
Code: [Select]
$ exiftool -DateTimeOriginal *.fff 
======== seq101.fff
Date/Time Original              : 2017:05:19 12:45:33.583-07:00
======== seq102.fff
Date/Time Original              : 2017:05:19 12:45:33.617-07:00
======== seq103.fff
Date/Time Original              : 2017:05:19 12:45:33.650-07:00
======== seq104.fff
Date/Time Original              : 2017:05:19 12:45:33.683-07:00
======== seq105.fff
Date/Time Original              : 2017:05:19 12:45:33.717-07:00
...
======== seq262.fff
Date/Time Original              : 2017:05:19 12:45:38.951-07:00
======== seq263.fff
Date/Time Original              : 2017:05:19 12:45:38.984-07:00
======== seq264.fff
Date/Time Original              : 2017:05:19 12:45:39.018-07:00
======== seq265.fff
Date/Time Original              : 2017:05:19 12:45:39.051-07:00
======== seq266.fff
Date/Time Original              : 2017:05:19 12:45:39.085-07:00
======== seq267.fff
Date/Time Original              : 2017:05:19 12:45:39.118-07:00
======== seq268.fff
Date/Time Original              : 2017:05:19 12:45:39.151-07:00

167/(39,151-33,583)=30 fps


check the last frame
Code: [Select]
$ exiftool seq267.fff
ExifTool Version Number         : 10.40
File Name                       : seq267.fff
Directory                       : .
File Size                       : 157 kB
File Modification Date/Time     : 2017:10:24 23:47:13+02:00
File Access Date/Time           : 2017:10:24 23:49:57+02:00
File Inode Change Date/Time     : 2017:10:24 23:47:13+02:00
File Permissions                : rw-r--r--
File Type                       : FLIR
File Type Extension             : fff
MIME Type                       : application/unknown
Creator Software                : RTP
Emissivity                      : 0.95
Object Distance                 : 1.00 m
Reflected Apparent Temperature  : 20.0 C
Atmospheric Temperature         : 20.0 C
IR Window Temperature           : 20.0 C
IR Window Transmission          : 1.00
Relative Humidity               : 50.0 %
Planck R1                       : 21546.203
Planck B                        : 1507.2
Planck F                        : 1
Atmospheric Trans Alpha 1       : 0.006569
Atmospheric Trans Alpha 2       : 0.012620
Atmospheric Trans Beta 1        : -0.002276
Atmospheric Trans Beta 2        : -0.006670
Atmospheric Trans X             : 1.900000
Camera Temperature Range Max    : 150.0 C
Camera Temperature Range Min    : -40.0 C
Camera Temperature Max Clip     : 160.0 C
Camera Temperature Min Clip     : -60.0 C
Camera Temperature Max Warn     : 150.0 C
Camera Temperature Min Warn     : -40.0 C
Camera Temperature Max Saturated: 180.0 C
Camera Temperature Min Saturated: -60.0 C
Camera Model                    : FLIR T1030sc 12__
Camera Part Number              : 72501-0201
Camera Serial Number            : 72501090
Camera Software                 : 33.0.0
Lens Model                      : FOL83
Lens Part Number                : T198942
Lens Serial Number              : 74200341
Field Of View                   : 11.9 deg
Filter Model                    :
Filter Part Number              :
Filter Serial Number            :
Planck O                        : -6331
Planck R2                       : 0.016229488
Raw Value Median                : 13782
Raw Value Range                 : 516
Date/Time Original              : 2017:05:19 12:45:39.118-07:00
Focus Step Count                : 65535
Focus Distance                  : 0.7 m
Raw Thermal Image Width         : 1024
Raw Thermal Image Height        : 768
Raw Thermal Image Type          : JPG
Raw Thermal Image               : (Binary data 156552 bytes, use -b option to extract)
Meas 1 Type                     : Spot
Meas 1 Params                   : 512 384
Meas 1 Label                    : 1
Palette Colors                  : 224
Above Color                     : 170 128 128
Below Color                     : 50 128 128
Overflow Color                  : 67 216 98
Underflow Color                 : 41 110 240
Isotherm 1 Color                : 100 128 128
Isotherm 2 Color                : 100 110 240
Palette Method                  : 0
Palette Stretch                 : 2
Palette File Name               : /FLIR/usr/etc/iron.pal
Palette Name                    : Iron
Palette                         : (Binary data 672 bytes, use -b option to extract)
GPS Map Datum                   : WGS84
Peak Spectral Sensitivity       : 9.5 um

tricky is the embedded thermal image, because "ff d8" are  the magic bytes of compressed JPG
Code: [Select]
$ exiftool -b -RawThermalImage seq267.fff  > 267.raw
Code: [Select]
$ hexdump -C -n 64  267.raw
00000000  ff d8 ff f7 00 0b 10 03  00 04 00 01 01 11 00 ff  |................|
00000010  f8 00 0d 01 ff ff 00 21  00 5c 01 37 00 40 ff da  |.......!.\.7.@..|
00000020  00 08 01 01 00 06 00 00  00 00 7a c2 c6 86 89 e5  |..........z.....|
00000030  4a d2 51 94 22 6a e6 a2  aa ba ee 73 6a 62 08 74  |J.Q."j.....sjb.t|

Imagemagick can't identify this jpg
Code: [Select]
identify: Unsupported marker type 0xf7

I think it's a Flir proprietary 16 bit compressed format

try to open the *.fff files with Flir Tools
« Last Edit: October 24, 2017, 10:13:48 pm by tomas123 »
 

Offline tomas123

  • Frequent Contributor
  • **
  • Posts: 832
  • Country: de
Re: CSQ file format
« Reply #4 on: October 24, 2017, 10:20:06 pm »
Code: [Select]
ff d8 ff f7

This is the lossless jpg format JPG-LS
http://www.digitalpreservation.gov/formats/fdd/fdd000151.shtml

Code: [Select]
$  ffmpeg -codecs | grep jpegls
ffmpeg version 3.4 Copyright (c) 2000-2017 the FFmpeg developers
...
 DEVILS jpegls               JPEG-LS

$ ffmpeg -f image2 -vcodec jpegls -i 267.raw  -f image2 -vcodec png 267.png
ffmpeg version 3.4 Copyright (c) 2000-2017 the FFmpeg developers
 
Input #0, image2, from '267.raw':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: jpegls, gray16le(bt470bg/unknown/unknown), 1024x768, lossless, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (jpegls (native) -> png (native))
Press [q] to stop, [?] for help
Output #0, image2, to '267.png':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: png, gray16be, 1024x768, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc57.107.100 png
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=0.21x   
video:629kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

$ identify 267.png
267.png PNG 1024x768 1024x768+0+0 16-bit Grayscale Gray 644587B 0.000u 0:00.009


now stretch to max/min
Code: [Select]
$ convert -auto-level 267.png 267.jpg
 :)
« Last Edit: October 24, 2017, 10:24:01 pm by tomas123 »
 

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #5 on: October 24, 2017, 11:09:02 pm »
Thank you Tomas!  Yes, this is what I couldn't figure out.  I was not aware of a lossless jpg format.  This was really helpful!   
 

Offline tomas123

  • Frequent Contributor
  • **
  • Posts: 832
  • Country: de
Re: CSQ file format
« Reply #6 on: October 25, 2017, 07:40:58 am »
I thought, that splitting is your first problem.
How did you split the stream into individual images?

PS: I wrote 2 years ago in the exiftool forum about the Flir jpg format in JPG-LS used in Flir Ultramax Images
http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,6763.msg33943.html?PHPSESSID=3tf2ulds7gepff5u9g9drpi095#msg33943
« Last Edit: October 25, 2017, 07:45:02 am by tomas123 »
 
The following users thanked this post: gtatters

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #7 on: October 25, 2017, 08:26:43 am »
Well, since I don't write Perl, I didn't split the images, I was just using a Hex editor and could follow some of your original messages back and forth with Phil Harvey to decipher some things.  I program in R and work in ImageJ which are not ideal for this purpose, but work for a non-programmer like myself. 

I hadn't got to the stage of splitting frames up, since I didn't know what the compressed format was anyway.  I could see the FFF header, but honestly wasn't sure where to go from there.

If you are ever interested in writing up an ImageJ plugin (mostly Java, I think), that could import FLIR SEQ, CSQ, and JPG, you would be doing the research community a huge favour!  Flir's research grade software is expensive (I have licenses for ResearchIR max) but with every new version they remove useful features from older versions.
 

Offline tomas123

  • Frequent Contributor
  • **
  • Posts: 832
  • Country: de
Re: CSQ file format
« Reply #8 on: October 25, 2017, 10:41:45 am »
Well, since I don't write Perl, I didn't split the images

you don't need some perl skills and it works also great with the portable perl version for windows

see my post here
http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,5279.msg25567.html?PHPSESSID=74a7ng9265gpj3knnt58ndbg06#msg25567
Code: [Select]
using:
Linux: ./split.pl 320x1.seq
portabe Windows: perl-5.18.0.1-32bit-portable\perl\bin\perl.exe -f split.pl SEQ1000.seq

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #9 on: October 25, 2017, 11:17:01 am »
Oh, I can run the scripts on my mac.  It's developing an easy to employ tool for non-computer savvy colleagues I am ultimately hoping to figure out.  ImageJ is the open source software for image analysis, and it would be ideal to have a direct importer. 
 

Offline Bud

  • Super Contributor
  • ***
  • Posts: 6877
  • Country: ca
Re: CSQ file format
« Reply #10 on: October 25, 2017, 02:19:07 pm »
The other useful plugin for ImageJ would be a SuperResolution one that takes a stack of images and combines in a super resolution image. I have a feeling it may be fairly simple but I do not have any Java skills at all.
Facebook-free life and Rigol-free shack.
 
The following users thanked this post: gtatters

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #11 on: October 26, 2017, 12:19:51 am »
Sorry to trouble you tomas, but what would be the syntax with ffmpeg for converting all the frames contained within one .raw file.  Your nice example above shows how to convert one image (frame 267), but when I try to expand this to a .raw file with all the fff data, I cannot seem to get ffmpeg to work.

I am trying this:
Code: [Select]
ffmpeg -f image2 -vcodec jpegls -i tempfile.raw -f image2 -vcodec png burst%02d.png
but get the following output with error:

Code: [Select]
ffmpeg version 3.3.2 Copyright (c) 2000-2017 the FFmpeg developers
  built with Apple LLVM version 8.1.0 (clang-802.0.38)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.3.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-libmp3lame --enable-libvpx --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-vda
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
Input #0, image2, from 'tempfile.raw':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: jpegls, gray16le(bt470bg/unknown/unknown), 1024x768, lossless, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (jpegls (native) -> png (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'burst%02d.png':
  Metadata:
    encoder         : Lavf57.71.100
    Stream #0:0: Video: png, gray16be, 1024x768, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc57.89.100 png
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=0.232x   
video:878kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
 

Offline -jeffB

  • Regular Contributor
  • *
  • Posts: 115
Re: CSQ file format
« Reply #12 on: October 26, 2017, 01:08:01 am »
If you are ever interested in writing up an ImageJ plugin (mostly Java, I think), that could import FLIR SEQ, CSQ, and JPG, you would be doing the research community a huge favour!

I don't get by this forum as much as I used to, and it's been ten years or so since I contributed to ImageJ, but this is interesting enough for me to drop a bookmark. :) I'm going to be tied up for the next week or so, but I'm intrigued at the notion of dusting off my ImageJ plug-in skills...
 
The following users thanked this post: gtatters

Offline Bud

  • Super Contributor
  • ***
  • Posts: 6877
  • Country: ca
Re: CSQ file format
« Reply #13 on: October 26, 2017, 04:33:34 am »
This is how I import .SEQ into ImageJ

File->Import->Raw

and use these settings (for 320x240 but I guess offsets should be same for other resolutions). Specify the required number of frames to import.



Here is someone describing his superresolution composition method using photoshop (scroll down to "Processing")

https://petapixel.com/2015/02/21/a-practical-guide-to-creating-superresolution-photos-with-photoshop/
Facebook-free life and Rigol-free shack.
 

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #14 on: October 26, 2017, 05:00:47 am »
I don't know how to write plugins for imagej, but I did write two macros to import SEQ files (converged on the same solution to Bud).

The first macro imports the SEQ into ImageJ (using defaults that work for me) and the second macro converts the raw into temperature, according to equations developed by tomas123 and Phil Harvey.  The first macro calls Exiftool and extracts the calibration constants.

But if someone can put this together as an ImageJ plugin (ie. faster performance than macro), I would be grateful.  Now I just need to get this to work for FLIR JPG and FLIR CSQ and I'm set. 

BTW, this code is still a work in progress, I only share it in response to jeffB who might dust off his plug-in skills!

First macro:

Code: [Select]
// offsetbyte is 1372 for SEQ files recorded to computer (works for at least two diff cameras)
// offsetbyte is 1540528 for SEQ files recorded to a FLIR SC660
// offsetbyte is 1542956 or 1540480 for SEQ files recorded to a FLIR SC640
// gapbytes is 3020 for direct SEQ recorded files
// gapbytes is 1424 for thermacam researcher pro captured seq files
var offsetbyte = 1372;
var gapbytes = 1424;
var nframes = 10000;
var imagewidth = 640;
var imageheight = 480;
var converttotemperature = 1;
var usevirtual = 1;
var minpix=1;
var maxpix=65535;

macro "Import FLIR SEQ" {
//Create Dialog Box
Dialog.create("Information for Thermal Video Import");
Dialog.addNumber("Offset Bytes", offsetbyte, 0, 8, "bytes");
Dialog.addNumber("Gaps Between Frames:", gapbytes, 0, 8, "bytes");
Dialog.addNumber("Number of Frames: ", nframes, 0, 8, "frames");
Dialog.addNumber("Image Width:", imagewidth, 0, 6, "pixels");
Dialog.addNumber("Image Height:", imageheight, 0, 6, "pixels");
Dialog.addCheckbox("Convert to Temperature on Import", converttotemperature);
Dialog.addCheckbox("Use Virtual Stack", usevirtual);
    Dialog.show();
   
//Define Variable for Import
offsetbyte = Dialog.getNumber();
gapbytes = Dialog.getNumber();
nframes = Dialog.getNumber();
imagewidth = Dialog.getNumber();
imageheight = Dialog.getNumber();
converttotemperature = Dialog.getCheckbox();
usevirtual = Dialog.getCheckbox();

filepath=File.openDialog("Select a File");
file=File.openAsString(filepath);
print("Loading: ", filepath);
print("\n");

run("Raw...", "open=filepath image=[16-bit Unsigned] width=imagewidth height=imageheight offset=offsetbyte number=nframes gap=gapbytes little-endian use=usevirtual");

OS=getInfo("os.name");

if(OS=="Mac OS X"){
flirvals=exec("/usr/local/bin/exiftool",  "-Planck*", "-*Emissivity", "-*Distance", "-*Temperature", "-*Transmission",  "-*Humidity", "-*Height", "-*Width", "-*Original", "-*Date",  filepath);
}

if(substring(OS, 0, 7)=="Windows"){
flirvals=exec("c:/Windows/exiftool.exe", "-Planck*", "-*Emissivity", "-*Distance", "-*Temperature", "-*Transmission",  "-*Humidity", "-*Height", "-*Width", "-*Original", "-*Date",  filepath);
print(flirvals);
}
        var PR1 = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Planck R1"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "Planck R1")) ));
var PB = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Planck B"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "Planck B")) ));
var PF = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Planck F"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "Planck F")) ));
var PO = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Planck O"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "Planck O")) ));
var PR2 = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Planck R2"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "Planck R2")) ));
var E = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Emissivity"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "Emissivity")) ));
var OD = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Object Distance"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "Object Distance")) ));
var RTemp = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Reflected Apparent Temperature"))+1, indexOf(flirvals, "C\n", indexOf(flirvals, "Reflected Apparent Temperature")) ));
var ATemp = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Atmospheric Temperature"))+1, indexOf(flirvals, "C\n", indexOf(flirvals, "Atmospheric Temperature")) ));
var IRWTemp = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "IR Window Temperature"))+1, indexOf(flirvals, "C\n", indexOf(flirvals, "IR Window Temperature")) ));
var IRT = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "IR Window Transmission"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "IR Window Transmission")) ));
var RH = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Relative Humidity"))+1, indexOf(flirvals, "%\n", indexOf(flirvals, "Relative Humidity")) ));
var imagewidth = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Width"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "Width")) ));
var imageheight = parseFloat(substring(flirvals, indexOf(flirvals, ":", indexOf(flirvals, "Height"))+1, indexOf(flirvals, "\n", indexOf(flirvals, "Height")) ));

print("Camera Calibration Constants:");
setFont("SansSerif", 12);
print("Planck R1: ", d2s(PR1,9));
print("Planck B: ", d2s(PB,9));
print("Planck F: ", d2s(PF,0));
print("Planck O: ", d2s(PO,0));
print("Planck R2: ", d2s(PR2,9));
print("Thermal Image Width: ", imagewidth);
print("Thermal Image Height: ", imageheight);

print("\n");
print("Default Object Parameters:");
print("Emissivity: ", d2s(E,2));
print("Object Distance: ", d2s(OD,2));
print("Reflected Apparent Temperature: ", d2s(RTemp,2));
print("Atmospheric Temperature: ", d2s(ATemp,2));
print("IR Window Temperature: ", d2s(IRWTemp,2));
print("IR Window Transmission: ", d2s(IRT,3));
print("Relative Humidity: ", d2s(RH,2));
print("\n");

Stack.getStatistics(count, mean, min, max, std);
var minpix=min;
var maxpix=max;
setMinAndMax(minpix, maxpix);

if(converttotemperature) {
run("Raw2Temp");
}

run("Calibration Bar...", "location=[Upper Right] fill=White label=Black number=5 decimal=1 font=12 zoom=1 bold overlay");
}


Second macro:

Code: [Select]

var PR1=21106.77;
var PR2=0.012545258;
var PB=1501;
var PF=1;
var PO=-7340;
var E = 0.95;
var OD = 1.0;
var RTemp = 20.0;
var ATemp = 20.0;
var IRWTemp = 20.0;
var IRT = 1.0;
var RH = 50.0;

macro "Raw2Temp" {
Dialog.create("Object Parameters to Calculate Temperature");
Dialog.addMessage("Camera Calibration Constants:");
Dialog.addNumber("Planck R1:", PR1, 2, 12, "unitless"); //21106.77 //21546.203
Dialog.addNumber("Planck R2:", PR2, 8, 12, "unitless"); //0.012545258 //0.016229488
Dialog.addNumber("Planck B:", PB, 0, 5, "unitless"); //1501 //1507.2
Dialog.addNumber("Planck F:", PF, 0, 2, "unitless");//1
    Dialog.addNumber("Planck O:", PO, 0, 5, "unitless"); //-7340 //-6331
    Dialog.addMessage("Object Parameters:");
    Dialog.addNumber("Object Emissivity:", E, 3, 6, "unitless");
    Dialog.addNumber("Object Distance:", OD, 1, 6, "m");
    Dialog.addNumber("Reflected Temperature (C):", RTemp, 2, 6, "C");
    Dialog.addNumber("Atmospheric Temperature (C):", ATemp, 2, 6, "C");
    Dialog.addNumber("Window Temperature (C):", IRWTemp, 2, 6, "C");
    Dialog.addNumber("Window Transmittance:", IRT, 3, 6, "unitless");
    Dialog.addNumber("Relative Humidity:", RH, 2, 6, "%");
Dialog.show();

var PR1 = Dialog.getNumber();
var PR2 = Dialog.getNumber();
var PB = Dialog.getNumber();
var PF = Dialog.getNumber();
var PO = Dialog.getNumber();
var E = Dialog.getNumber();
var OD = Dialog.getNumber();
var RTemp = Dialog.getNumber();
var ATemp = Dialog.getNumber();
var IRWTemp = Dialog.getNumber();
var IRT = Dialog.getNumber();
var RH = Dialog.getNumber();

ATA1 = 0.006569; //Atmospheric Trans Alpha 1
ATA2 = 0.012620; //Atmospheric Trans Alpha 2
ATB1 = -0.002276; //Atmospheric Trans Beta 1
ATB2 = -0.006670; //Atmospheric Trans Beta 2
ATX =  1.900000; //Atmospheric Trans X

emisswind = 1- IRT;
  reflwind = 0 // anti-reflective coating on window
  h2o = (RH/100)*exp(1.5587+0.06939*(ATemp)-0.00027816*(ATemp)^2+0.00000068455*(ATemp)^3); // converts relative humidity into water vapour pressure (I think in units mmHg)
  tau1 = ATX*exp(-sqrt(OD/2)*(ATA1+ATB1*sqrt(h2o)))+(1-ATX)*exp(-sqrt(OD/2)*(ATA2+ATB2*sqrt(h2o))); // atmos transmittance from object to window
  tau2 = ATX*exp(-sqrt(OD/2)*(ATA1+ATB1*sqrt(h2o)))+(1-ATX)*exp(-sqrt(OD/2)*(ATA2+ATB2*sqrt(h2o))); // atmos transmittance from window to camera

rawrefl1= PR1/(PR2*(exp(PB/(RTemp+273.15))-PF))-PO   // radiance reflecting off the object before the window
rawrefl1attn = (1-E)/E*rawrefl1   // attn = the attenuated radiance (in raw units)

rawatm1 = PR1/(PR2*(exp(PB/(ATemp+273.15))-PF))-PO // radiance from the atmosphere (before the window)
rawatm1attn = (1-tau1)/E/tau1*rawatm1 // attn = the attenuated radiance (in raw units)

rawwind = PR1/(PR2*(exp(PB/(IRWTemp+273.15))-PF))-PO
rawwindattn = emisswind/E/tau1/IRT*rawwind

rawrefl2 = PR1/(PR2*(exp(PB/(RTemp+273.15))-PF))-PO   
rawrefl2attn = reflwind/E/tau1/IRT*rawrefl2

rawatm2 = PR1/(PR2*(exp(PB/(ATemp+273.15))-PF))-PO
rawatm2attn = (1-tau2)/E/tau1/IRT/tau2*rawatm2

rawsubtract = (rawatm1attn + rawatm2attn + rawwindattn + rawrefl1attn + rawrefl2attn)
rawdivisor = E*tau1*IRT*tau2

run("32-bit", "stack");
run("Macro...", "code=v=" +PB+ "/log(" +PR1+ "/(" +PR2+ "*(v/" +rawdivisor+ "-" +rawsubtract+ "+" +PO+ "))+" +PF+ ")-273.15 stack");

mintemp=PB/log(PR1/(PR2*(minpix/rawdivisor-rawsubtract+PO))+PF)-273.15;
maxtemp=PB/log(PR1/(PR2*(maxpix/rawdivisor-rawsubtract+PO))+PF)-273.15;
setMinAndMax(mintemp, maxtemp);
}
« Last Edit: October 26, 2017, 06:25:06 am by gtatters »
 

Offline Bud

  • Super Contributor
  • ***
  • Posts: 6877
  • Country: ca
Re: CSQ file format
« Reply #15 on: October 26, 2017, 05:59:15 am »
Should the two macros be in two individual files?
Facebook-free life and Rigol-free shack.
 

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #16 on: October 26, 2017, 06:14:07 am »
Yes, sorry about that.  I just placed them into my Startup Macro so that they are available when you boot up imageJ.

If you goto Plugins-->Macros-->Startup Macros, you can paste those at the bottom, hit save, and restart ImageJ.  Then they will both show up in the menu item Plugins-->Macros.

You may need to customise a few things, I just shared after seeing your post with the import-->raw option (which I used within the macro). 
 

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #17 on: November 03, 2017, 05:39:12 am »
Any chance you know how to extract (using Exiftool) all frame times from a .seq file without having to split first into .fff files?

For example:
Code: [Select]
exiftool -DateTimeOriginal sample.seq
will only yield one value,

Code: [Select]
Date/Time Original              : 2016:06:06 15:41:09.884+01:00

but if I split into multiple .fff files, using tomas123's code

Code: [Select]
perl split.pl sample.seq
and

Code: [Select]
exiftool -DateTimeOriginal sample.seq
produces:
Code: [Select]
======== frame00001.fff
Date/Time Original              : 2016:06:06 15:41:09.884+01:00
======== frame00002.fff
Date/Time Original              : 2016:06:06 15:41:10.912+01:00
======== frame00003.fff
Date/Time Original              : 2016:06:06 15:41:11.944+01:00
======== frame00004.fff
Date/Time Original              : 2016:06:06 15:41:12.968+01:00
======== frame00005.fff
Date/Time Original              : 2016:06:06 15:41:13.996+01:00
======== frame00006.fff
Date/Time Original              : 2016:06:06 15:41:14.924+01:00
======== frame00007.fff
Date/Time Original              : 2016:06:06 15:41:15.952+01:00
======== frame00008.fff
Date/Time Original              : 2016:06:06 15:41:16.980+01:00
    8 image files read

I have some large .seq and .csq files that take ages to extract .fff files.

Any suggestions?

 

Offline Bud

  • Super Contributor
  • ***
  • Posts: 6877
  • Country: ca
Re: CSQ file format
« Reply #18 on: November 03, 2017, 06:35:17 am »
Custom programming I guess... not through scripting.
Facebook-free life and Rigol-free shack.
 
The following users thanked this post: gtatters

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #19 on: November 03, 2017, 08:07:57 am »
Sorry to trouble you tomas, but what would be the syntax with ffmpeg for converting all the frames contained within one .raw file.  Your nice example above shows how to convert one image (frame 267), but when I try to expand this to a .raw file with all the fff data, I cannot seem to get ffmpeg to work.

I am trying this:
Code: [Select]
ffmpeg -f image2 -vcodec jpegls -i tempfile.raw -f image2 -vcodec png burst%02d.png
but get the following output with error:
...



Nevermind!  I figured it out, by exporting a series of extracted tiff files to an avi:

Code: [Select]
ffmpeg -f image2 -vcodec tiff -r 30 -i 'frame%05d.tiff' -pix_fmt gray16be -vcodec png file.avi
 

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #20 on: February 08, 2018, 02:03:10 pm »
I had a follow up question/problem with the split function in perl.  I have been running this successfully for months on a Mac/Unix system/environment.  I am able to split out all frames to .fff files. 

Recently I tried this on a Windows 10 machine.  Installed Exiftool (had to turn off windows defender!).  Installed Activestate perl and ffmpeg all successfully.

But when I run the split script on a .csq file, the .fff files created in windows are different size from that created in unix.  See attachments created under the two different environments.  Usually it creates the correct number of files on both operating systems, suggesting the search pattern is fine.

The culprit appears to be how \x0A is interpreted in Unix vs. Windows.  Windows inserts \x0D\x0A when writing the .fff files, creating a mess.  Then when i pass the .fff files to exiftool, they are not seen as FLIR format files. 

"binmode out" is set in the perl script and I've read that this is important for working in windows environments. 

Does anyone have any suggestions?  I am trying to help out a friend who only uses windows.  :(

Here is the perl script:

Code: [Select]
#!/usr/bin/perl
undef $/;
$_ = <>;
$n = 0;
$pat="\x46\x46\x46\x00";
my $directory = "temp";
 unless(-e $directory or mkdir $directory) {
        die "Unable to create $directory\n";
    }
# FLIR camera E40         
# Flir Tools (comment out)
#$pat = "\x46\x46\x46\x00";
for $content (split(/(?=$pat)/)) {
        open(OUT, ">temp/frame" . sprintf("%05d",++$n) . ".fff");
        binmode OUT;
        print OUT $content;
        close(OUT);
}
 

Offline tomas123

  • Frequent Contributor
  • **
  • Posts: 832
  • Country: de
Re: CSQ file format
« Reply #21 on: February 11, 2018, 04:55:01 pm »
I see in my notes:
in 2013 I used successful the code above under windows with the (old) Version 5.18 32 Bit of portable strawberry Perl for Windows

http://strawberryperl.com/releases.html

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #22 on: February 12, 2018, 01:35:53 am »
Thanks for the suggestion.  I tried it out, and there is some improvement, but still some issues, and these are enough to cause exiftool to fail, so I am stuck again.  :(

Using strawberry-perl-portable 32 bit on windows 10 (the only OS I can try this on).

In this example (see image), 3 instances of x0A are replaced with x0D.  In the non-portable perl installed on windows, the number of these errors is a few hundred in one file.

Any suggestions?
 

Offline tomas123

  • Frequent Contributor
  • **
  • Posts: 832
  • Country: de
Re: CSQ file format
« Reply #23 on: February 12, 2018, 01:46:16 pm »
sorry for your trouble, but I haven't yet enough time, to code another solution for you.

The perl-code is very simple, so you can it fast convert to another script language (java, php etc.)

I fast entry for decoding binaries with php is my Flir panorama script :
Code: [Select]
php splitjpg.php -i IR_0556.jpg -r stitch-ps16.png -o pano1pslink: https://www.eevblog.com/forum/thermal-imaging/flir-e4-thermal-imaging-camera-teardown/msg348715/#msg348715

from an old post:
- hint for windows user for using php as a portable version without some installations:
    download from http://windows.php.net/download/#php-5.4 the current version 5.4.xy
    extract zip to a folder php-5.4.xy-nts-Win32-VC9-x86
    run > php-5.4.xy-nts-Win32-VC9-x86\php.exe flir.php ... options...

please post your solution here

Offline gtattersTopic starter

  • Contributor
  • Posts: 44
Re: CSQ file format
« Reply #24 on: February 12, 2018, 02:05:38 pm »
No worries, Thomas, I wasn't expecting you to create a new solution!  I was perhaps hoping that someone might know why the perl code is so OS dependent, since it works fine on mac and linux. 

As for other coding languages, I'm a biologist, not a computer scientist, so progress will be slow.  But I'll have a look at the php approach to see if I can work it out. 

Thank you for pointing me toward that.  I'm keeping track of my working solutions at my github site also:

https://github.com/gtatters/Thermimage
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf