Thermal Expert- Raspberry Pi

[kiranjayaraj]

Hello guys,
I have been reading this blog for a few months for information related to my university project. We are building a canopy temperature sensor using IR camera and we did an implementation with a Seek camera on Raspberry Pi.
I recently got hold of a new Thermal Expert Q1, which I am now trying to integrate to my Raspberry Pi and I'm not having the best times with it. I see that there are a lot of posts about the Q1, but does anybody have a solid way to have the TE interact with the Raspberry Pi. I installed their SDK and tried their instructions on running it on the Pi, but it doesn't seem to work at the moment. I can get to those details in trailing posts, but I was wondering if someone had a simpler test code to use the TE (in Python perhaps)? I am also new to Raspberry Pi and linux environments, so appologies in advance if the question seems wrong. (Noob Alert)
Thanks in advance for your responses.
Kiran

[IwuzBornanerd]

kiranjayaraj,

I'm trying to reply to your PMs but the system says your inbox is full so I'll put a reply here & hope you see it.

You asked if it is normal to not see a /dev/video device for the thermal camera on a Pi and that you have seen a FLIRone being viewed on a webcam app.

Yes, that is normal for a thermal camera because there are no standard V4L (or VFW in Windows) drivers for thermal cameras.  The reason you can plug in most webcams and view the output in various different camera programs is that there is a standard (Video for Windows & Video 4 Linux) that specifies a set of standard function calls that all compliant programs will use to get information from the cameras and each camera vendor provides a driver that accepts and responds to that standard set of function calls.  Thermal cameras are not cheap and common enough for that to have happened yet.  They might get there in a few years, though, especially with the Chinese cameras coming out.  That shared library file provided by I3 is the sort of thing that is needed and then someone would need to either make thermal camera viewer programs or add the capability to existing webcam software.  Around 12 years ago or more you could buy webcams that you had to use their special viewers for because the Windows standard was not fully adopted.  It has been fewer years for Linux, and there might still be some webcams that don't work under Linux because the vendor does not want to give out proprietary information.

I suspect that what you saw for the FLIRone is what the guys developed on the " Question about FLIR One for Android" topic(?)  What they did is essentially make a V4L driver for the FLRIone--exactly what Ben321 wanted except for Linux instead of Windows.   

But they used a package called "vloopback", which as far as I know is not officially supported anymore.

Seek does not provide any sort of driver, dll, or shared library like I3 does so there is nothing but hacked information to use with Seek.

[mahony]

Hi, late reply but I played a bit with the Raspi Zero and the TE-Q1.
Attached is some very basic code to get images from the Thermal Expert Q1 - for the moment this is really only getting 50 frames as they are from the camera and dumping them to disk. It's just a main.cpp plus simple makefile. The code was tested on a desktop machine (and written) and a Pi Zero W - compiles and runs on both without issues BUT on the Pi I had to run it as sudo for whatever reason. Only requirements are make, gcc/g++ compiler and libusb-1.0 (I used sudo apt-get install libusb-1.0-0-dev).

For any further image processing (dead-pixel removal, factory shutterless correction, manual NUC and so on) you either have to write your own stuff or wait until I get on with this and post an update.

This is NOT using the supplied TE-SDK but I probably also build a version using it at some point just to compare.

[Vipitis]

what kind of framerate are expectable from a very very basic Pi or Pi Zero?

[mahony]

Hi vipitis,
I did not actually measure the time but it just took a few seconds to grab the 50 frames. I assume you can easily achieve the default 8.5 Hz even on a PI Zero (W). I mean the data rate is really low (384x296x2bytesx8.5Hz = ~2MBytes/s) and even the basic image corrections computations should be doable - it is just a simple lookup and multiply-add per pixel (for simple gain-offset) or a 4th order Polynom per pixel (factory shutterless gain-offset correction).

I can probably do some timing tests this week. I want to extend the code anyway to at least replace the dead pixels and maybe also do the 4th-order gain-offset correction plus show the final image in a simple OpenGL window (not sure if OpenGL will do a the PI Zero - I never tried so far).

EDIT: just tried it on the Pi Zero - and I am pretty unimpressed! So far I did not get significantly above 3fps with dead pixel replacing AND full shutterless correction - no matter if the frames where just read or also dumped to disk and also independent from execution in either pure console or from the desktop. I did not yet get the frames displayed in a GL window on the Pi - seems like I have choosen an improper pixel format or something.

I may do some tests on a Pi 3B+ too but that is not the platform(s) I am really interested in Linux-wise so do not expect too much effort here - I have already skrewed the afternoon getting the GL libs installed on the Pi just to find out my code is not running as is ... 

[mahony]

Finally got some GL  code running on the Pi Zero W and Pi3B (so 1.2 GHz NOT 1.4) and adapted my code for the Thermal Expert - it is now running smooth on both.

The 3B achieves somewhere around 8.5Hz at roughly 40% CPU load while the Pi Zero W finally did slightly above 4 fps but almost maxed out on CPU. I enabled the full MKS OpenGL driver on both and had to install some GL libraries on the Zero.

The code is attached plus some actual screenshot of the app running on the 3B. What is still missing is some code cleanup (i.e. move stuff to appropriate classes) and reading the dead pixel map, gains and offsets from camera instead of reading them from disk.

[Vipitis]

Is your imager limited to 8.5hz or do you have the unlocked 30hz version?

I am just trying to figure out if a DIY handheld monocular would be possible with a small but open computer.

[IwuzBornanerd]

...BUT on the Pi I had to run it as sudo for whatever reason.

In order to NOT need root (sudo) privileges you need a file in the /etc/udev/rules.d/ directory that contains a line like

Code: [Select]

SUBSYSTEM=="usb", ATTRS{idVendor}=="I3vendorID", ATTRS{idProduct}=="TEQ1ID", GROUP="plugdev"Where I3vendorID is the vendor ID number for the camera and TEQ!ID is the ID number for the camera.  You can get the numbers from the command lsusb -v

I don't think the file name matters, but this file was somehow auto-generated with the file name 50-seekthermal-usb.rules for my Seek on my laptop running Debian "Wheezy".  It has not been automagiacally done on any of my Pi's.

And yes you do need at least libusb-1.0-0-dev (and the library itself), older versions won't work.

Does the Zero allow you to enable the REAL MKS driver now, or is it the "fake" MKS?  "Raspbian Jessie" did not work with it (at all), I think.

If you are better at multi-threading than I am you might be able to get some improvement if you put the USB transfer code in a separate thread and also the display writing in a separate thread.  I have to have the display stuff (using dispmanx video core commands) in a separate thread on the Zero in order to get 8fps off a plain non-pro Seek.  Since there is only one CPU core I have not tried putting the USB stuff in a separate thread.  I see no chance of getting any where near 30fps on a Pi though.

[tmbinc]

I have a custom core (I _will_ follow with details at some point) on a Raspberry PI. My setup is an OpenVizsla board (random FPGA board that I had lying around) receiving LVDS data (FPD-Link II; an interesting topic in itself), feeding that over an FT2232H, and displaying that on a RaspPi (3B, non-plus). The FPGA packetizes the LVDS data and reduces the data slightly (FPD-Link II is 24 bit/pixel payload, but 16-bit are sufficient; 14-bit of thermal data + 2 bits for sync). The RaspPi receives the data (via scanlime's "fastftdi", https://github.com/openvizsla/ov_ftdi/blob/master/software/host/fastftdi.c ), and does the pixel processing (all in software).

I then either output that in a eglfs SDL2 image, or via eglfs qt. (qt because I wanted to provide a UI, but I struggled there...).

The data is around 324x256 with 30fps, and I can display that in realtime.

Some notes:

- Raspberry Pi's USB (dwc2) is _a piece of crap_. (If you look into how it actually works - it's a horror story. One of the issues is that you need to service the stupid USB controller for every microframe. You literally need a full CPU core just for that - and in fact, that's what's for example Windows 10 IoT on Raspberry Pi is doing. ).
- In Linux, just use multiple cores. I have one USB receive thread, one pixel processing thread, one display thread. They get nicely scheduled across the quad cores.
- My pixel processing is plain C code. No fancy shader. I receive a uint16_t framebuffer[2][325][256] (i.e. double-buffering, so I can fill the framebuffer with USB data while the pixel processing thread is running on it; otherwise I would have to serialize or sync the operations - but hey, memory is plenty!), I do histogram equalization (see https://stackoverflow.com/questions/34126272/histogram-equalization-without-opencv ), which requires 1 pass over the image to build the histogram, 1 pass over the histogram (16384 entries) to build a 14-bit-to-RGB LUT, and then one more pass over the pixel data. Since for a typical image only few distinct pixel values are used, the 16384 * 4 bytes LUT cache performance isn't that bad.) The RGB values I derive from a pre-calculated palette with, say, 1024 entries. (So it's really 14-bit-to-10-bit-to-RGB but that's fair enough)
- The end result is a 32-bit framebuffer that I either poke directly to /dev/fb, or submit via whatever GUI library I'm in the mood for.

The most critical path is the USB bandwidth. 324x256x30 is roughly 2.5 MPixel/s, which works with 16 bit per pixel (~5MByte/s), but not with 24bpp. I prototyped the setup with a Zynq-based ZynqBerry, which worked well, but was too expensive in the end.

I can see to share my code but it's super crappy, and I don't have time to work on it right now.

[mahony]

@Vipitis: I am not yet aware of a 30Hz Q1 (unfortunately ) - so it is actually limited to pretty close to 9 Hz. I think the 3B should be capable of handling close to 30 fps from an M1 (with 240x180 Pixels). And even the Zero should be able to get to 8/9 fps with the M1 as the amout of data to be processed is less than half of the Q1. I tested my Code with the V1 yesterday on my desktop so I could try it on the Pi too - but I expect to get noticably less than 30 fps with the 640x480 resolution.

Just another example for the processing: I have a Teensy 3.6 running at just 180 MHz that happily grabs the data from a 160x120 Lepton3 at 26 fps (thats the data rate you have to handle: 1 good frame + 2 junk frames) displays it to a 160x128 SPI TFT (8.7 fps) and also dumps the raw data to SD card (once per second). 

As @tmbinc and @IwuzBornanerd pointed out multithreading may significantly improve Performance on the Pi 3 - most likely not on the Zero as it's a single-core.
320x240 at 30 Hz is already quite demanding for the Pi but sounds absolutely reasonable. On a medium i5 processor PC the 640x480x30 Hz cause little more than 30% CPU load so I would assume the 1.2 GHz Pi might be able to get at least 10-15 fps here - probably even more with some streamlined / multithreaded Code.

I may try this on the Pi when the code gets to this point but my primary target is a Jetson TX1 ... so plenty of power here. ;-)

@IwuzBornarnerd: thanks for pointing me to the rights issue - I assumed something like this but did not really dig into the issue as my main PC is not having it.
Regarding the GL Driver on the PI zero: I just activated it through raspi-config but did not actually check which Driver is really used on the zero. I used a new Image of Raspbian Stretch - maybe there are some changes but I am not sure.

[jglomas86]

Thanks for the code Mahoni, you have gone beyond what thermal expert does. 
Regarding the .bin outputs and .rad
What fileformat / post-processing do they need to get to an image?
Thanks,
JL

[mahony]

Hi,
thanks, the code is far from 'finished' but I am lacking time somehow at the moment for all the fun stuff with thermal imaging, coding and so on.
The .bins are basically a raw dump of the 16-bit pixel data, the .rad format is very similar but has a small header of 28 bytes that holds mostly image width, height, bytes per pixel, and pixel type (8, 16, 32, 64 bit / int or floating point) plus some magic number for id purposes.

Attached is a a C# app that can display those files (plus a lot of image processing stuff that is somehow useful to me) and save the data as displayed to .bmp/.jpg/.tiff or .png files. NOTE: there are two .zip files due to file size limits. The app should run without the OpenTK dlls but there is a menu point "3D Ansicht" which needs them. (Yes the menus are mostly German - I have to re-write it at some point and that is on the TODOs …)

[jeng]

@Mahony,
About libusbTEQ1GL, exactly what I need,
But is it possible to pipe this to gstreamer?
I have no X11 server running on my pi zero.
I want to pipe this in gstreamer an then transfer it via WiFi broadcast.
https://github.com/rodizio1/EZ-WifiBroadcast/wiki

[Ultrapurple]

Slightly off topic, I know, but In case it's of any interest, Wi-Fi broadcast is built in to the Therm-App native driver and at least one other third-party app (ThermApp Plus). It runs on Android, rather than the various Linux platforms such as the Pi, but many Android stick-type computers will work fine. The main requirement is that the stick must support USB OTG (not just be able to support a USB keyboard/mouse etc).

If I recall correctly, ThermApp Plus establishes an independent Wi-Fi hotspot (sorry, I can't remember the exact terminology or details; it may possibly be Wi-Fi Direct). I am not sure but I think the stock app requires an internet connection, but is viewable anywhere in the world.

PM me if you want any more info and I'll see what I can dig up.

[mahony]

Hi,
I assume this should be possible but I never used either gstreamer nor ez-wifibroadcast. You will most likely have to create a compressed videostream from the raw 'bitmaps' that are displayed. I assume that is what gstreamer is for, if not ffmpeg should do. But I got no idea how to put together all the rest. Data rate is states as <12MBit/s so uncompressed might do to but you have to check your tools how to get what you need.

On the other hand I also interfaced a Lepton3 from the Pi3 some time ago and build a very basic(!) UDP-based server on the Pi that streamed the raw data to some host PC via local WLAN. This worked ok-ish but would need some work to be more reliable/stable.

[jeng]

Hi,
I assume this should be possible but I never used either gstreamer nor ez-wifibroadcast. You will most likely have to create a compressed videostream from the raw 'bitmaps' that are displayed.

Ok, that's what I need, but have a problem.
Im not a developer.
So this would be the difficult part for me.


Gesendet von meinem SM-G950F mit Tapatalk

[jeng]

Slightly off topic, I know, but In case it's of any interest, Wi-Fi broadcast is built in to the Therm-App native driver and at least one other third-party app (ThermApp Plus). It runs on Android, rather than the various Linux platforms such as the Pi, but many Android stick-type computers will work fine. The main requirement is that the stick must support USB OTG (not just be able to support a USB keyboard/mouse etc).

If I recall correctly, ThermApp Plus establishes an independent Wi-Fi hotspot (sorry, I can't remember the exact terminology or details; it may possibly be Wi-Fi Direct). I am not sure but I think the stock app requires an internet connection, but is viewable anywhere in the world.

PM me if you want any more info and I'll see what I can dig up.

Hi,
I can give it a try,
I really need this cam working.
The thermal app you are talking, is the TE-Q1 compatible?

If so,
Have a nanopi a64 board here, I can flash android v6.0 on it to test.

http://linux-sunxi.org/FriendlyARM_NanoPi_A64

Gesendet von meinem SM-G950F mit Tapatalk

[Ultrapurple]

I really need this cam working.
The thermal app you are talking, is the TE-Q1 compatible?

Unfortunately the Therm-App is a different device from the TE-Q1, made by a different company. As far as I know there is no common driver. mahony and others know a lot more about I do about running dongle-type cameras on Linux systems and are better placed to advise on your specific issues. The only reason I mentioned the Therm-App camera range was in case someone else is reading this thread in the future looking for a similar, off-the-shelf answer.

Of course there's nothing to stop you selling one camera and buying another but that generally works out fairly expensive. Engineering a solution to your current setup is likely to be a more satisfactory way to go - but of course I don't know your budget.

I should point out that I am very biased towards Therm-App because they are my main area of practical experience and I like the way the cameras work for me. I am not professionally associated in any way with Therm-App or the parent company Opgal - they do not give me any payment or other benefits).

[Requiem for a Dreamcast]

Finally got some GL  code running on the Pi Zero W and Pi3B (so 1.2 GHz NOT 1.4) and adapted my code for the Thermal Expert - it is now running smooth on both.

The 3B achieves somewhere around 8.5Hz at roughly 40% CPU load while the Pi Zero W finally did slightly above 4 fps but almost maxed out on CPU. I enabled the full MKS OpenGL driver on both and had to install some GL libraries on the Zero.

The code is attached plus some actual screenshot of the app running on the 3B. What is still missing is some code cleanup (i.e. move stuff to appropriate classes) and reading the dead pixel map, gains and offsets from camera instead of reading them from disk.

Mahoney, great job with this code! I have adapted your code to work with the M1, and it runs well enough on the Pi 3B. However, since I don't know how to get the dead pixel map or the gain and offsets from the camera, the values are not scaled properly (the dead pixels throw off the range). I have managed to make my own maps more or less, but they are a hack job. How did you get your proper maps?

[mahony]

Hi Requiem,
to be honest I read the shutterless data (gains, offsets, dead pixels plus some calibartion constants) via my more functional C# app and just copied the data over.
But it is relatively easy to read all of the data from the first 26 frames (probably less frames on the M1?) after the camera is connected. I think I described it somewhere in the Thermal Expert Q1 thread - it is more or less the same for the Q1 and the V1 and I think also for the M1 as far as I could see in the disassembled Java code. Most likely the calibration constants are in a different place - thats the main difference between Q1 and V1 from my experience.

I can describe it to you if required or maybe point you to the existing description in the TE thread.
At some point I am planning on porting all functionality of my C# code to C++ for use on the Raspi, Jetson Nano and Jetson TX1 - mostly for performance reasons. ;-)

[Requiem for a Dreamcast]

Thanks Mahony, I've gone through your posts and found the ones where you explain what the first frames do. Info is a bit all over a few threads so it can be hard to find everything in one place, but following your posts pointed me to the right one. I'll see if I can figure it out when I get some more free time. Cheers.

[mahony]

Late update:
As I got a PN question about the software I though it might be helpful to some if I finally post some updated code.
This will now also read all relevant data from camera flash (only after each reconnect to the PC! TBD: find and include flash read request command) and store them to disk.
I also added full radiometric support using all the required calibration data.

I did not test it under Linux (just VS2017 for today) but it should still compile and run. If compiled in windows you can click to the image to get DL and temperature for this pixel along with current image min/max and FPA temperature.

From here on it should be easy for anybody to build whatever is needed from this camera on almost any platform (even a perfoming Android Phone version should be doable -> TBD for the V1).

[agh768]

will take a look. maybe i can get it running in my thermal spectrum app

[mahony]

@123Dante123: your inbox is full, so I answer here:

You don't need nvcc - this is the CUDA C compiler from NVIDIA which is only required if compiling with

Code: [Select]

#define CUDA activated in the code.
I use this on the NVIDIA Jetson Nano to speed up image correction.

You should be able to compile the program with this on the command line or in a  bash script (build.sh in .zip):

Code: [Select]

g++ -c -o3 -Wall -Wextra -Wpedantic main.cpp
g++ -c -o3 -Wall -Wextra -Wpedantic ThermalExpert.cpp
g++ -o main main.o ThermalExpert.o -lusb-1.0 -lX11 -lGL -lGLU -ldl
This will just compile the 2 source files to object and then link everthing to the executable called 'main'.
You should not have the TE drivers installed on the system for this - or at least deactivate them. I am using the generic libusb-1.0 for direct raw access to the camera!
I do all the image processing that is done in the driver the same way but it seems to be more efficient. ;-)

Edit:
regarding required libs: on a bare Linux system (i.e. ubuntu) you may need to install the dev libs for GL, GLU and libusb-1.0 using:

Code: [Select]

sudo apt-get install libgl1-mesa-dev
sudo apt-get install libglu1-mesa-dev
sudo apt-get install libusb-1.0-0-dev
On the RPI you may need to also install the X11 libs - I can rember what was exactly needed on the RPI3 so you may need to google or post the errors you get.
It should be something like:

Code: [Select]

sudo apt-get install libx11-dev

[agh768]

tried to compile the vs solution. i'm using VS2019 and installed cuda v11.0 and libusb1.0.23
but when starting it is recognizing the camera but the TE-Viewer only shows some rubbish and the window locks up. Not sure if its because i changed the references or using VS2019