Author Topic: OpenIRV. ISC0901B0 (Autoliv NV3, FLIR E4/5/6/8) based opensource thermal camera  (Read 111740 times)

0 Members and 1 Guest are viewing this topic.

Offline LesioQ

  • Regular Contributor
  • *
  • Posts: 66
  • Country: pl
  • Every king should be naked.
NV3 lens' clear aperture is 15mm. Objective is of a two-lens design.
Lots of cost-saving on NV3, including Ge protective window thickness being 1/2 NV2.
 

Offline tisher

  • Contributor
  • Posts: 10
  • Country: pl
so 'only' need to find focal length for first lens  :-+
 

Offline mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 13694
  • Country: gb
    • Mike's Electric Stuff
I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.
A few years ago, after I did a teardown video on one of these, someone emailed me, with some info they'd reverse-engineered, but didn't want to publish due to ITAR concerns.
The main point was that they had found that the original FPGA used a Microblaze core, and it was easy to find the code image in the flash memory, and after disassembling with IDA, they managed to reverse-engineer the handshake protocol used to authenticate the camera with the control box.
There might be some useful insights in to how any sensor-unique data is stored.
 
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline Ismsanmar

  • Contributor
  • Posts: 43
  • Country: es
I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.
A few years ago, after I did a teardown video on one of these, someone emailed me, with some info they'd reverse-engineered, but didn't want to publish due to ITAR concerns.
The main point was that they had found that the original FPGA used a Microblaze core, and it was easy to find the code image in the flash memory, and after disassembling with IDA, they managed to reverse-engineer the handshake protocol used to authenticate the camera with the control box.
There might be some useful insights in to how any sensor-unique data is stored.

Are you talking about this?:
https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/
 

Offline LesioQ

  • Regular Contributor
  • *
  • Posts: 66
  • Country: pl
  • Every king should be naked.
so 'only' need to find focal length for first lens  :-+
What's the purpose of it's focal length ? For f-stop calculation You need effective f.l. of complete objective.
 

Offline mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 13694
  • Country: gb
    • Mike's Electric Stuff

Had a look at the schematic. Debouncing of the buttons will be done in VHDL via a filter in the FPGA? In the microcontroller unit it is an annoying task to debounce something.
Buttons pull-ups and debouncing capacitors are at the P-board (peripheral). Of course, there is additional debouncing logic in the HDL design. Agree, filtering buttons by internal mcu is a poor idea.


Debouncing entirely in software is trivially easy - you simply sample the buttons every 20-40mS or so.

« Last Edit: October 18, 2020, 11:13:27 am by mikeselectricstuff »
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline tmbinc

  • Regular Contributor
  • *
  • Posts: 249
I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.
A few years ago, after I did a teardown video on one of these, someone emailed me, with some info they'd reverse-engineered, but didn't want to publish due to ITAR concerns.
The main point was that they had found that the original FPGA used a Microblaze core, and it was easy to find the code image in the flash memory, and after disassembling with IDA, they managed to reverse-engineer the handshake protocol used to authenticate the camera with the control box.
There might be some useful insights in to how any sensor-unique data is stored.
Hi Mike, yes, that was me. I've since then published my findings - in slightly redacted form - in https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/.

However, my question specifically is a.) about the NV3 family, and b.) more specifcally the FLIR PathFindIR II, and how it differs from the automotive NV3 that we've all seen.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hello everyone, some updates...

Sorry again for this delay... but this 2020 year is a nightmare in absolutely every aspect of life.
But I try to do my best to keep developing the project no matter what happens.

Back on project:

1. As the hardware development is almost done, I have ordered electronic parts for the new hardware revision last monday, hope to get them in 2-3 weeks (yes...too long, but there is no any other way to speed up this process).

2. I have finally finished all shematics and routed all PCBs, including recomendations that I received from you. The whole set of schematics is published, chekout my repo.
PCB board production time is about 1 week, but I'm going to ordered new boards next week to get both boards and components simultaneously. If you have any suggestions according to newest schematics, please let me know, this week is the last chance to make any changes.
One more link to the repo: https://github.com/OVGN/OpenIRV/tree/master/docs/schematics

3. All firmware and HDL sources will be published along with porting to new hardware. As soon as I get sure that new hardware is working well, I will start beta production for testers and developers.
« Last Edit: October 23, 2020, 07:15:48 pm by VGN »
 

Offline dmendesf

  • Frequent Contributor
  • **
  • Posts: 318
  • Country: br
Congratulations, your Altium skills are top notch.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hi! Some updates.

Good news, I received most part of active components, i.e. power supply ICs and the FPGA! Now waiting for some passive smd stuff from another supplier. New PCBs were ordered several days ago. Hope to start assembling everything in a week or a little later.

Meanwhile, I decided to make some improvements to the focusing system. First, I have modified the motor mount. Earlier I noticed that motor rigid connection to the board is not a good idea, as motor vibrations sound is amplified by the focus PCB. I decided to use a special silicone cover between the motor and the motor mount. I haven't printed this parts yet, so let me know if you have ideas of impromevents.

The second modification is the focus motor itself. Somebody noticed that focus adjustment is really slow. Yes, this is true. This is because I have selected a motor with very high gear ration and low rpm (1:136 @240RPM) as I was not sure if this tiny motor can apply enough force to move the lens. There is another type of this motor with lower gear ration and higher RPM (1:26 @1200RPM). I haven't ordered it yet, but...I decided to modify one of my motors. If you take it apart, you will find 3 stages of tiny planetary gear. I removed one the stages, shortened the motor housing and put all back together. Now it is running much faster! Checkout the difference at video:
https://youtu.be/GuIjXj8-yEI

Congratulations, your Altium skills are top notch.
Thanks, though there so many thing to learn)

« Last Edit: November 07, 2020, 02:53:06 am by VGN »
 
The following users thanked this post: ajw107

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 53
  • Country: de
New focus speed is for my purpose now fast enough.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
New focus speed is for my purpose now fast enough.
One more video of focusing right from the thermal camera. This is not autofocus yet, I'm adjusting the focus manually and not accurate enough sometimes. The autofocus will do this job much better that me.

Youtube collapces video quality too much, that's why download this video before viewing, it is only ~39MB.
https://drive.google.com/file/d/1_MNzznl6It1Ix80HIM0NVI9G4iD7iLla/view?usp=sharing
« Last Edit: November 07, 2020, 12:40:54 pm by VGN »
 
The following users thanked this post: ddrl46, Cat, rockwell, horstmannhid

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
While waiting for PCBs and components, I decided to reveal some future plans.

ISC0901 is a pretty good 336x256 60FPS sensor. This characteristics is enougth for quite a lot of different applications.
But I have one a bit crazy idea. I would like to significially increase the resolution. As you know, the whole hardware was
designed in a way to support any other sensor (I mean, configurable power lines, a lot of high performance IOs, RAM, etc.)
But unfortunatelly this is really hard to find cheap high resolution sensor...

If you look at the SEM photos of LWIR array, you could see a quite interesting pixel structure. Note, this is a photo of the ISC0601, not ISC0901.
The main difference is the pixel pitch. ISC0601 has 25um pixel pitch, while ISC0901's pixel pitch looks like to be 17um.
I could not find SEM photos of the ISC0901 sensor pixel array, except one photo of the beam holding "via", that proves somehow that ISC0901 has the same
pixel pattern as ISC0601.

If you have or saw somewhere the SEM photos of ISC0901 pixel array, please share it here, that will be very helpful!

The pink area is actually the active pixel, this is a resistor that is supported by two special thin beams, that allow to make
thermal insulation of the pixel from the die and other pixels. The radiation comming from the lens heats up each pixel.
The resistance of the each pixel changes according to the radiation level. This resistance value is calculated by internal ADC,
so that's how we get the pixel value.

Due to technology limitations, the active pixel area is only about 21% of the whole FPA area. There are huge gapes between pixels, that are not doing any useful work.
So simply shifting the array for a 1/2 of pixel pitch value in X and Y direction, we could increase the resolution from 336x256 to 672x512.

But this is also not a limit yet, as making more accurate steps (1/4 or 1/8 pixel pitch) along with special deconvolution algorithms it is possible
to increase the resolution even more! The last attached photo shows the result of x5 super resolution image reconstruction, based on 20 low resolution frames. Looks quite impressive for me.

I'm not sure, but I hope to get a megapixel camera this way))
« Last Edit: November 14, 2020, 03:28:01 pm by VGN »
 
The following users thanked this post: ajw107

Offline zrq

  • Frequent Contributor
  • **
  • Posts: 273
  • Country: 00
If someone in the EU have a faulty sensor, please send it to me. I'll be happy to get some SEM images. (I'll try my best  ;) to persuade my advisor to allow me to do that)
 
The following users thanked this post: VGN

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
If someone in the EU have a faulty sensor, please send it to me. I'll be happy to get some SEM images. (I'll try my best  ;) to persuade my advisor to allow me to do that)
That will be really nice to get SEM photos, but not mandatory, as we know the most important value - the pixel pitch of 17um. FLIR’s Tau2 camera core has 17 μm pixel pitch and we know that it is based on ISC0901.

Probably knowing dimentions of the pixel's active area and gaps around it will be helpful for proper enlarged image calculation. I think this the only reason to sacrify single lwir core and get SEM photos.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Now two words about how I'm going to implement this pixel shifting.

Well, we are talking about micrometer precision level of displacement. I know that there is a trick of using handheld motion effect,
that allows to get this small sensor displacements (Fluke implemented this feature on some of their thermal imagers).
But don't like it at all, as handheld motion based super resolution (SR) is not accurate enough and also will not work in case of tripod mounting.

I'm going to use piezo actuators along with a custom XY-stage.

You can find a lot of info about piezo actuators and the physics of  inverse piezoelectric effect. In two words this is crystal that can enlarge
for some small value if you apply some voltage. Piezo actuator behaves like a common capcacitor, and displacement value is equivalent to the charge level.
The dependency between the voltage and actual displacenent value is linear. It is possible to get even nm accuracy if desired.

ISC0901 has 17um pixel pitch, so we should select an actuator with a displacement value > pixel_pitch/2. I have selected AE0203D18H18DF, that has
max displacement level of 19.0 ±2.0um at 150V. Actuator dimentions are 18x3x2mm. The generated displacement force just blows my mind, it is about 200N (yes, two hundred).

I made some sketches of the XY stage and also some simulations. The thickness of the moving platform beams is about 0.8mm. If made of 6061 aluminium we need to
apply only 10N to displace the platform for about 16um, while the actuator can generate 200N.

Note that the movement level at attached gifs is amplified a lot, this is not the real value, as 18um displacement value is so small, that this is simply impossible to see it with an unaided eye.

I have never developed anything using such kind of XY-stages and piezo actuators, so suggestions and critics are welcome.



Advantages:

1. It is possible to significially increase the resolution. The gaps between active pixel area are so big, that pitch/2 shift will give a clear x4 resolution. More accurate shifting steps along with deconvolution image processing will increase the resolution even more, I hope to get a megapixel.
2. The construction will be fully backward compatible with whole other design, including focusing board.
3. This XY stage can be used with any other sensors, if attached over special adapter. New LWIR cores have even lower pixel pitch of 12um and we can still increase the resolution.

Disadvantages:

1. The piezo actuator, this piece of... shi ceramics... costs around 100$. Yes, it is even more expensive that the FPGA. And...we need two of them for X and Y axis.  :scared:
2. We will have to deal with high DC voltages under 150V, well this is a bit more dangerous than usual. There are special ICs for driving actuators with integrated dc-dc boosters that allow to get this high voltage.
3. Not sure, but probably the whole construction will be a bit more fragile, expecially because of the ceramic actuators, though I'm sure you wouldn't let it fall down)
« Last Edit: November 15, 2020, 01:51:13 pm by VGN »
 

Offline Vipitis

  • Frequent Contributor
  • **
  • Posts: 867
  • Country: de
  • aspiring thermal photography enthusiast
Pixel shifting can be found in Sony cameras and Hasselblad. There are many thread in this subforum that touch on the idea and some did tests. Pixel shifting is not viable for video, but a slow photomode for high resolution stills would be a great feature I would love to have.
The claims about active area vs Pixel pitch Sounds a little odd. We know that Leonardo/DRS has a special pixel design that is a lot more area efficient than the one shown here, using "micro umbrellas". It might be the sole reason why they managed to 10μm somewhat useable.

Also a thought: while you shift the sensor, why not include a whole 5 axis platform for in body stabilizion? Sony uses that for their pixel shifting as well. And they do capture like 12 individual frames on the a7RIV to get 120MP and no interpolation.

Do you hope to shift by perfectly 1/2 the pixel pitch or would 1.5 also work?  If you go beyond 4x this would require getting 0.33,0.66 or even 0.25, -0.25 too. While it might seem that perfect precision makes this sound possible - the real world has limitations in holding perfectly still. So a potential photomode could also include stacking of same pixels shifted by 1 to reduce noise.


There is great development done in this technology for surveillance satellites, so it's very much inaccessible.
 

Online Fraser

  • Super Contributor
  • ***
  • Posts: 13148
  • Country: gb
In case this paper is of use/interest.....

Vibration induced Resonance and the effects of G forces in microbolometer pixels ......

https://www.jvejournals.com/article/15829/pdf

Fraser
« Last Edit: November 15, 2020, 03:11:41 pm by Fraser »
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 
The following users thanked this post: zrq, VGN

Offline mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 13694
  • Country: gb
    • Mike's Electric Stuff
So simply shifting the array for a 1/2 of pixel pitch value in X and Y direction, we could increase the resolution from 336x256 to 672x512.
Bear in mind you may need a much better ( more expensive) lens to get improved resolution. If it is the case that it's only sampling a fairly small percentage of the sensor area, then you would see noticeable aliasing artifacts (e.g. jaggies on straight edges) unless the existing optics system has been designed to slightly blur the image to antialias it. It may also be the case that there is some post-processing going on as well to reduce aliasing effects.
I'm sure there is some scope to improve resolution by moving the sensor and applying some clever maths, but I'm not convinced the results would be spectacular.
You can see this effect in this video where I was under-sampling a video camera :
https://youtu.be/rQYByorpoFk?t=869
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 
The following users thanked this post: VGN

Offline SilverSolder

  • Super Contributor
  • ***
  • Posts: 6126
  • Country: 00

Wouldn't it have to be a pretty terrible lens to not be able to resolve  336x256, for example?
 

Offline mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 13694
  • Country: gb
    • Mike's Electric Stuff

Wouldn't it have to be a pretty terrible lens to not be able to resolve  336x256, for example?
Yes, but it is probably engineered to that resolution, giving just the right amount of blur to avoid aliasing artifacts at that resolution
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 
The following users thanked this post: SilverSolder

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Pixel shifting can be found in Sony cameras and Hasselblad. There are many thread in this subforum that touch on the idea and some did tests. Pixel shifting is not viable for video, but a slow photomode for high resolution stills would be a great feature I would love to have.

Yes, but as I know pixel shifting in common cameras is mostly used to breaktrough the limitation on Bayer filter, i.e. shifting the sensor you can get true R,G,B values for each pixel, so there is no need to make debayering, as we already have all color components. I don't really know, why it is called super resolution, as it is not that, we don't get more pixels:

Pixel shifting is not viable for video, but a slow photomode for high resolution stills would be a great feature I would love to have.
True, but piezo actuators are extremely fast. So theoretically it is possible to shoot a video, though the FPS will be n^2 times lower, where n is a factor of resolution enlargement.

The claims about active area vs Pixel pitch Sounds a little odd. We know that Leonardo/DRS has a special pixel design that is a lot more area efficient than the one shown here, using "micro umbrellas". It might be the sole reason why they managed to 10μm somewhat useable.
That was news to me! So it turns out that each "micro umbrella" just heats up the same VOx film, which is placed under it, supported by the same know thermal isolation pattern? And we have almost no pixel gaps. This is really incredible! Checkout attachments.

Also a thought: while you shift the sensor, why not include a whole 5 axis platform for in body stabilizion? Sony uses that for their pixel shifting as well. And they do capture like 12 individual frames on the a7RIV to get 120MP and no interpolation.
That is going to be too complex and expensive...Moreover, no idea what to do with 5 axis.

Do you hope to shift by perfectly 1/2 the pixel pitch or would 1.5 also work?  If you go beyond 4x this would require getting 0.33,0.66 or even 0.25, -0.25 too. While it might seem that perfect precision makes this sound possible - the real world has limitations in holding perfectly still. So a potential photomode could also include stacking of same pixels shifted by 1 to reduce noise.

I hope to shift with much smaller steps that 1/2. Anyway this is a matter of real experiments, as I expect some limitations and problems. I also should find some way for step calibration.
« Last Edit: November 15, 2020, 04:02:36 pm by VGN »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Vibration induced Resonance and the effects of G forces in microbolometer pixels ......
Thanks, Fraser. That should be taken in account, but I think that vibration frequency caused by pixel shifting actuators will be far away from the sensor's pixel resonance frequency.

Bear in mind you may need a much better ( more expensive) lens to get improved resolution.
Thanks, Mike, I didn't thought about that. Well, that may be a huge problem. I'm not really good with optics, but original FLIR E8 lens is much worse that this one, but it was also designed for the right same ISC0901 sensor.

Do you have any ideas how to check this lens capabilities?
 

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1514
  • Country: be
1. The piezo actuator, this piece of... shi ceramics... costs around 100$. Yes, it is even more expensive that the FPGA. And...we need two of them for X and Y axis.  :scared:

I would give these a try: https://nl.aliexpress.com/item/32874018830.html?spm=a2g0o.productlist.0.0.2b033905e3vloB&algo_pvid=23e2f0b8-e665-4619-bd80-5c8e64b3aa5c&algo_expid=23e2f0b8-e665-4619-bd80-5c8e64b3aa5c-11&btsid=0b0a555c16054735796882807e75f6&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_

180µm for 100V, should be around 18µm for 10V, and as you are 3D printing your housing and absolute accuracy is not required for this application, I think these would be sufficient.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I would give these a try: ...
180µm for 100V, should be around 18µm for 10V, and as you are 3D printing your housing and absolute accuracy is not required for this application, I think these would be sufficient.
Hm... the characteristics look quite fantastic for this dimentions. Have you ever used this parts?

For example, both different well known manufacturers, and the parts with common dimentions have commom characteristics:
Kemet's AE0203D18H18DF from here: https://ru.mouser.com/datasheet/2/212/1/KEM_P0101_AE-1518874.pdf
PI's P-882.51 from here: https://static.pi-usa.us/fileadmin/user_upload/physik_instrumente/files/datasheets/P-882-Datasheet.pdf

I have absolutely no experience with piezo actuators, but chinese ones look really strange. Maybe I'm wrong, not sure this is truth. We need anyone how can prove that.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf