Seek Compact Pro frame rate investigation

[Spirit532]

I recently received a late-2017 Compact Pro(NIB), and after playing with it for a while(including at the USB level) it's puzzling me as to the actual frame rate the device is capable of outputting.
From my own tests, I am able to achieve about 10-18fps intermittently, both on PC and on an Android phone(Xiaomi Redmi Note 4, plenty of processing power).
Here's an average video stream I get:


However, this video shows that on an iPhone 7, the same device outputs nearly full-rate video.


Logically, a question arises from this:
Since the frame rate is already way past the U.S. ITAR and EAR export regulation limits for slow-rate cameras, what is limiting the Android version to 12-18 fps, even when connected to a really powerful computer(via a separate, otherwise empty USB3.0 controller), and how is it different from the iPhone version which comes close to full-rate?

It appears to be quite different in terms of the USB protocol as well - I have not been able to turn on the raw data mode as you could previously with the Compact and XR.

If anyone would like to help figure this out, a dump of the communications between the iPhone and the iPhone version of the Compact Pro would be greatly appreciated.
Any input is welcome.

To calm a few people that may be nervous because of this discussion: The devices are already export-controlled and output >9Hz, this is not violating any laws. It doesn't matter whether it's 9.1 or 30.
Also, to those that know how to bypass the Compact and XR's 9Hz cap: Please, do not post it here or give any hints. Keep it legal.

[tonykids]

according to jetdail’s post https://www.eevblog.com/forum/thermal-imaging/thermal-imaging-gallery/75/
I find the way to display the framerate on android(su)
the config file is in /data/data/com.tryriansystems.SeekThermal/shared_prefs/com.tryriansystems.SeekThermal_preferences.xml
enable the "export_frame_rate_allowed_key" then you can see the framerate


I captured the usb communication between an android simulator and seek pro.the frame rate in the simulator is 6-9.5hz and the communication is jut the same as i posted herehttps://www.eevblog.com/forum/thermal-imaging/windows-software-for-seek-pro-and-initial-commands/
It's about 90-100ms for seek pro to transfer one frame,so all the windows apps that support seek pro now has a frame rate <11.
 but my old android phone can reach 16hz

unzip the SeekThermal_2.1.1 and find something...
Maybe the command on x86(x64) cpu in the seek thermal app is different from it on an arm cpu
Hope someone has a USB protocol analyzer

[Spirit532]

but my old android phone can reach 16hz

I'm not sure if it's a USB limitation on the PC or if it's a separate command set that lets you get a higher frame rate.
Sadly most of the low-level USB stuff appears to be done through libSeekware, not the native app itself.

After some investigation I found this bit of code:

Code: [Select]

    public String f1() {
        switch (f2()) {
            case COMPACT:
                return "Compact";
            case COMPACT_XR:
                return "CompactXR";
            case COMPACT_FF:
                return "Compact FF";
            case COMPACT_XR_FF:
                return "CompactXR FF";
            case COMPACT_PRO:
                return "CompactPRO";
            case COMPACT_PRO_XR:
                return "CompactPRO XR";
            case COMPACT_PRO_FF:
                return "CompactPRO FF";
            case COMPACT_PRO_XR_FF:
                return "CompactPRO XR FF";
            default:
                return "Unknown";
        }
    }
But following the code up, I can't figure out what it does with these.

[Fraser]

I have had nothing to do with the Seek cameras for a long time but something to keep in mind is that "frame rate" is an often misused term.

Under the various controls and regulations, the <9fps cap for simple licensing refers to the update rate of thermal image data and NOT that delivered to the user by the display system.

That is to say, a 9fps camera can, and often does, present the user with a display update rate well in excess of 9fps. 25fps and 30fps are common in order to feed a LCD panel or external monitor at a decent refresh rate.

I have a thermal camera that actually refreshes the thermal image at 120fps....... but the LCD display is refreshed at 30fps. That was just pure marketing spec manipulation to make the camera appear more capable. It was already a very capable camera core running at 60fps and down sampled to 30fps for the LCD display. This is the reverse of the usual 9fps situation but it proves that frame rate can mean much, and nothing, depending upon how it is presented.

So in summary, there is absolutely no reason why the microbolometer core frame rate need be the same as that presented to the user, be it via a display device or a data stream to a host system that further processes the data. 9fps at the core output, presented to the user LCD display at 30fps is relatively simple to achieve. This is why the true frame rates on <9fps cameras are in fact lesser rates in order to easily convert them to 30fps NTSC or 25fps PAL frame rates. Frame rate restricted NTSC compatible cameras run at 7.5fps (30/4) and PAL compatible cameras run at 8.33fps (25/3).

You have to ask yourself " Am I seeing "X" FPS of changing thermal data, or is it just the output of a conversion process that only truly provides a maximum of 9 unique frames per second ?

Food for thought

Fraser

[Spirit532]

You have to ask yourself " Am I seeing "X" FPS of changing thermal data, or is it just the output of a conversion process that only truly provides a maximum of 9 unique frames per second ?
Food for thought

None of this has anything to do with the issue at hand though, you can clearly see the Pro is producing actual, changing frames at a rate higher than 9 Hz, even in my video.
We know for certain both the 206 and 320 Seek sensors are producing 30Hz internally and can output 30 via USB, but for some reason something is slowing it down with the Pro.

[Fraser]

Oh I see.... different situation. Fair enough 

[Spirit532]

Small update: It appears that there are two versions of the Compact Pro - one marked UQ-EAA, and one marked UQ-AAA. I'm assuming E stands for Export and A for America, but I can't confirm the meaning of it beyond that, because I've seen units marked "FF" and UQ-EAA at the same time.

My unit is not marked with the "FF" symbol, and it identifies as a "Compact PRO" by the app. The product code is UQ-EAA, but it's still marked "Subject to US EAR Export Regulations".

Does anyone have a unit that is identified as a "Compact PRO FF", or has the "FF" symbol on the back of the body?

[Fraser]

Even a <9fps thermal camera that contains an A-Si or VoX microbolometer of certain pixel count is liable to the import/export regulations aplicable in the country of origin or resale. The restrictions are just less onerous than those applied to high frame rate cameras in many cases.

A <9fps camera still cannot legaly be exported to countries deemed inappropriate for such technology. NK and Iran being two that come easily to mind. Exemptions can be applied for, but on a case by case basis.

The SEEK will have to comply with US export regulations or else it would get contacted pretty quick by the BIS. The fact that greater than 9fps is reported to have been achieved from these cameras is interesting but the devil is in the detail. They may still comply with BIS requirements. Pixel count and true thermal image frame rate are assessed when BIS consider permitting general export under exemptions and extant policy. I do not see how an American company like SEEK could breach such regulations without BIS action against them.

It is an interesting topic but not one that I have, or will dig deeper into. Good luck with your investigations. It may be that SEEK limit frame rate within the host App and clever people can release a higher frame rate. Whether that complies with BIS requirements is for the lawyers. If the EAA version contains a frame rate limiter in the actual camera dongle, that may have satisfied the BIS. It would be interesting to know what the difference is as the differing marking surely must be for a purpose.

All the best with your investigation of the camera and its software 

Fraser

[Spirit532]

This topic is strictly for discussing performance differences in the >9 Hz-capable Pro, not about hacking the Compact and Compact XRs to break the 9Hz limit.
It's 320x240, so it's still under the 111kpx limit, but legally it shouldn't even matter because the Pro is already faster than 9 Hz.

[Fraser]

As stated, I am out of touch with the SEEK cameras after being so disappointed with them. I had not spotted that you are STRICTLY discussing a dongle that is intended to produce >15fps. I only knew the original models that were <9fps.

Why do you not just approach SEEK and ask them why the frame rate of the Android Version is inferior to that of the iPhone version. It would cut out your guess work and the need to profile the system data processing path and budget.

I may be able to give you hint though as I already know a little about why....... when designing an application for an Apple iPhone, the developer deals with a set of known platform and interface characteristics, both in hardware and software. Developers like Apple products for this reason. Present the developers with a requirement to write an app for an Android device and see them worried. Android is, as you well know, a multi platform operating system that runs on many different hardware platforms. The differences in said hardware platforms make it very hard for the developers to optimise the app for greatest efficiency. On the iOS platform they tweak the app to provide the very best performance. On the Android platform the developers are generally very happy if the app even runs correctly on a large cross section of platforms from differing manufacturers. Android can be a total nightmare when it comes the fine tuning and squeezing maximum performance out of the app that has to drive external hardware. In many cases, the developers find platforms, in this case phones, on which they cannot achieve app stability. These go on the 'not compatible' list. It does not matter how wonderful or powerful the phone is, if the designer cannot get the app to work well on it, the phone is not a good choice.

A simple case of a fine tuned and capable dedicated app for the very well defined iOS platform Vs a generic app build designed to work 'well enough' with an acceptable range of android platforms. Fine tuning ? Dream on ! The Android App is inferior to that produced for the single platform iOS product.

Writing your own Android app is always an option, but you can discover that the SDK contains a less than ideal or sprightly driver for the camera. The reason ? It has to remain compatible with the large cross section of Android platforms on which it will run.

How do I know this to be true and 'real World' and not just some crazy theory ? Simple, I spoke at length with the people who developed the FLIR One G2 and they faced the same issues. The FLIR iOS app is more efficient than the Android app as a result. The iOS app produces a higher frame rate than the Android app even though the camera hardware is equally capable. For info, FLIR developers had to choose a single phone model for which they would try to tune the Android app so that it compared well against the iOS version when tested and reviewed. They chose the Samsung S5 mobile phone (released 2014) as the, then, equivalent to the iPhone in terms of processing power. The iPhone still achieved better frame rates though and they could not bring the Android app up to the performance of the iOS version. The FLIR One G2 Android runs at its best performance on a Samsung S5 and likely its more recent versions. SEEK may also be able to advise which phone they chose to develop the app on. Then again, they may not have bothered with fine tuning for one model and kept things generic.

So there you go, that is the answer to "why" you are seeing frame rate variances. As to how you can squeeze better performance out of your SEEK setup ? Well that is a little challenge for you to engage on, if desired. Much depends upon how helpful SEEK are with regards to release of a decent SDK  Good luck on that front !

My apologies for reducing the signal to noise ratio in this thread. I will not comment further.

Fraser

[Spirit532]

None of this matters when you're connected to a PC using the motherboard's USB3.0 capable hub without any other devices connected to it.
They've made it pretty clear that the SDK is no longer a possibility, since even the developer program is gone now.

[Micz]

Hello, this is my first post and I am waiting for deliver seek compact pro. Have you checked hz on other android smartphone ? I am gimnazjum use it with oneplus 5 do i will report it. I wonder how you  connect camera with computer ? Maybe connection is the key ? My oneplus 5 is usb-c slot and I need adapter to connect it and there are many adapters which doesnt work. Local seller told me that its need to be a powerful phone to work good. Sorry for my english :-)

[tonykids]

I tested the seek pro on my 3 different phones
phone 1: Snapdragon 801+2G RAM-> 12~15fps ,can record video correctly
phone 2: MTK MT6735+2G RAM->       14-18fps ,only the fist 1~2 second in the recorded video is visible 
phone 3: Snapdragon 652+ 4G RAM-> 14~16FPS,only the fist 1~2 second in the recorded video is visible 
phone 2 is the least powerful one but it perform the best
By the way,all of my 3 adapters(gifts when i bought something) work well......

[Micz]

What cables / adapters do you use to connect the computer ?

[tonykids]

What cables / adapters do you use to connect the computer ?

I made it myself
It's really simple,a USB type A plug and microusb receptacle, match and connect the VCC D+ D- GND to each other

For the extension  cable on the phone ,connect the ID pin to GND

[Spirit532]

It would be helpful to know the model number of your devices for a start(whether it's UQ-EAA or UQ-AAA).

[tonykids]

It would be helpful to know the model number of your devices for a start(whether it's UQ-EAA or UQ-AAA).

Mine is UQ-EAA ,However,it's just a sticker. under the sticker it says:UQ-AAA

[Spirit532]

Same on mine!
I think they just use the same product box and put a sticker on top to indicate a different model.

But now we know the EAA model(which is apparently Export) is not limited in frame rate, which was one of my concerns.

[IwuzBornanerd]

I have been thinking about this matter myself as the one I obtained last week only produces 9fps on my 2.4GHz laptop consuming <8% of CPU time.  To add to your confusion, the model number on the box is UQ-AAAX (no sticker) and the box was sealed with 2 round "Export control" stickers.  I don't have a phone to use it with, but I tried it on a relative's phone yesterday.  I have no idea what the frame rate was (or what the phone was) but it looked a bit smoother than my 9fps.  And I think the image was not as clean as with my code.

Regarding the export rules I have been thinking that if the camera defaults to 9fps and there has to be a certain command issued from the software in order to enable it to run faster then it amounts to the same situation as the non-pro versions AND therefore any software that enables it to run faster would be controlled by the EAR because there is a specific line item in the regulations for software that defeats the 9fps limitation of a camera.  This would also explain why the "developer program" is gone as stated above.  Any 3rd party software would have to comply with EAR requirements just as Seek is having to do.  If I'm correct about this we can't talk about code to make the pro run at "FF" on this board even if the camera is sold as being able to run that fast, because it requires software to tell it to run that fast.

The Seek user guide says that registration is required to use the camera, but the app has a "do it later" option, so I was running it on the relative's phone without registering.  I wonder if registering is required to "unlock" the faster frame rate, but since I could not tell the frame rate for sure I didn't bother trying that while I had the phone.  I also wondered if once the camera was registered maybe something was stored on the camera to tell the software to allow FF.

I have not tried to determine how much actual time the USB transfer of a frame takes, but as I was working to get the pro running on my laptop I found that it takes 13 and sometimes 14 bulk transfer requests to get an entire frame.  The code I started with for the non-pro version (which I got from this board) used 4 fixed length transfers and this has ALWAYS worked for me on the non-pros.  I was forced to "do it right" with the pro & have it keep asking for however many bytes were remaining until it got all of them, and this is working well.  Maybe the way they keep it at 9fps is by feeding it setup commands that make it run the USB slow.

Interestingly, though, I see a lot more 9's in my frame count than I see with the non-pro cameras.

[Spirit532]

I have no idea what the frame rate was (or what the phone was) but it looked a bit smoother than my 9fps. And I think the image was not as clean as with my code.

It's the same for my device. I get around 10-ish frames per second on the PC, around 12-14 on the phone.

Regarding the export rules I have been thinking that if the camera defaults to 9fps and there has to be a certain command issued from the software in order to enable it to run faster then it amounts to the same situation as the non-pro versions AND therefore any software that enables it to run faster would be controlled by the EAR because there is a specific line item in the regulations for software that defeats the 9fps limitation of a camera.

This is a non-issue, certainly. I purchased mine from a retailer, shipped without warnings or paperwork, and the default app runs it at over 9Hz. It's jerky, but it does hit ~11-12, from estimations by eye - definitely faster than the ~10 I'm getting on PC(see first video). I'll unlock the debug features in a bit and see the actual frame rate I'm getting.
Not to mention the 30Hz mode of the non-PRO versions was never meant to leave the factory, and it never did - it was discovered by blind poking.

The Seek user guide says that registration is required to use the camera, but the app has a "do it later" option, so I was running it on the relative's phone without registering.  I wonder if registering is required to "unlock" the faster frame rate, but since I could not tell the frame rate for sure I didn't bother trying that while I had the phone.  I also wondered if once the camera was registered maybe something was stored on the camera to tell the software to allow FF.

Used it for a while without registering - I see no difference after(to get rid of the nag), or after the signing out. GPS was turned off, location was set to "United States" which is the default. Logged in, got rid of the nag, signed out, still no change in original performance.

Maybe the way they keep it at 9fps is by feeding it setup commands that make it run the USB slow.

Could be - does it enumerate as FS or HS? What's the polling rate?
I am a bit weary of trying to omit various commands without knowing what they do. If you'd like to take that risk, I would start with the new commands specific to the PRO.

[tonykids]

Amazing!
When i was playing the seek pro with my new PC(G4560+4G RAM + SSD +win7),i noticed that the frame count increased extremely fast than usual
So I captured the communication and find the seek pro was transferring a frame within 60ms or less
But I did nothing to the code
Maybe we just need a more powerful processor?

[Spirit532]

Maybe we just need a more powerful processor?

I have a high-end gaming machine that can render FHD video in near-realtime, so I don't think that's the issue.

[IwuzBornanerd]

You mentioned polling rate.  I captured some data transfer on an oscilloscope (photo attached) which shows, as @tonykids found, that it takes about 50ms or more to get a frame and then my code apparently spends another 60ms processing it, which pretty well explains the 9fps.  What I can find about USB polling rate is that it can be 125, 250, 500, or 1000Hz; but everything I find about setting that only applies to HID devices.  Apparently my bus is set to 250Hz If I could set it to 1000Hz I could presumably get 13-14fps and then it would be my code to blame for the slow speed.  Taking a whopping (and ridiculous) 50-60ms just to get a frame off the camera doesn't leave much processor time for making images at 15+fps.

Another thing when running on a phone, I've heard that phones can have a lot of crap running in the background consuming cpu time so that might be slowing the camera down on some phones.

[Spirit532]

So after enabling the frame rate indicator, it shows 9Hz at startup, dips to 5.6, and then stabilizes at 8.5-8.2.
I conclude that either the indicator is completely false and lying, or my eys are deceiving me, because what I see is definitely not consistent with the indicator.

You mentioned polling rate.  I captured some data transfer on an oscilloscope (photo attached) which shows, as @tonykids found, that it takes about 50ms or more to get a frame and then my code apparently spends another 60ms processing it, which pretty well explains the 9fps.

Perhaps multithreading is the solution here, for the PC? Get the data as fast as possible, measure that rate.

Another thing when running on a phone, I've heard that phones can have a lot of crap running in the background consuming cpu time so that might be slowing the camera down on some phones.

You're not wrong there, but my phone is rather powerful and it still chokes with no processing on, which is odd.
It could be that there are different, undocumented versions of the Pro that are limited to 9Hz, but all the evidence is counter to that.
There also may be some configuration bytes we're either missing or unknowingly adding that may be hindering the performance.

Just in case anyone wants to poke around the source code, sere's the latest Android app decompiled (47MB):
http://i.spirit.re/com.tyriansystems.SeekThermal_source.zip

To get started, look for "SUBI" in the contents of the files. Some juicy stuff pops up.

[IwuzBornanerd]

The 60ms gap between the frame data blocks in my photo above is driven by the camera not my software.  I turned off all my image correction math & the delay did not get any smaller, so I tried adding a delay in my code.  I could add 30ms of delay before seeing an increase in that gap, and dropping the 9th frame/sec.  Adding a 40ms delay added about 10ms to the gap and dropped the frame rate to 7-8fps.  So my code is actually only taking about 30ms to do its work and the other 30ms is what is left over of the delay the camera puts in there. 

I also checked the data stream from the non-pro XR and the gaps between the 4 data blocks are twice what they are between the 13 data blocks from the pro using the same program, so that delay is also from the camera and not the USB polling rate.

I am thus convinced that this camera is defaulting to 9fps even though it has the "FF" logo printed on its back.  I figure it must need either something written to memory on the camera or specific commands from the software to permit the higher frame rate.  I'd like to see some O-scope pictures of the data stream with a camera on a phone & see what all is different.