Author Topic: OpenIRV. ISC0901B0 (Autoliv NV3, FLIR E4/5/6/8) based opensource thermal camera  (Read 3748 times)

0 Members and 1 Guest are viewing this topic.

Offline joe-c

  • Frequent Contributor
  • **
  • Posts: 329
  • Country: de
    • Joe-c.de
its a really awesome project.
I am very interested.

currently, I work on an open-source solution too, but its more an interface of an existing Camera, not a completely new device.

I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.

anyway... really nice. Thanks for show it.
best regards
FLIR E4 hardware mods 
Freeware Thermal Analysis Software: ThermoVision_Joe-C
Freeware Remote Software for some FLIR Cameras: Ex Tools
 

Offline Ismsanmar

  • Contributor
  • Posts: 21
  • Country: es
VGN, incredible work. Another one here interested in the possibility of adapt your solution for the NV2 ISC0601 sensor. I'm another one with one of those lying around.


I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.

The problem with that camera is that only goes at 15fps on line detection mode
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
its a really awesome project.
currently, I work on an open-source solution too, but its more an interface of an existing Camera, not a completely new device.
I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.
Thank you!
No there is no visual camera, but I thought about that. You know, I held Flir EX in my hands once. And I can't say that MSX technology impressed me a lot. MSX cannot help when FPA's resolution is low itself or when you cannot change the lens focus. Also image alignment is not possible at short distances. Looks like MSX brings more engineering headache than benefits. Are there a lot of people using MSX? Maybe you have some arguments that can change my mind?

VGN, incredible work. Another one here interested in the possibility of adapt your solution for the NV2 ISC0601 sensor. I'm another one with one of those lying around.
Thanks!
Yes, I think it's possible to adapt this camera for NV2 sensor. But I don't have it, and know completely nothig about ISC0601 sensor. If you have this camera, you could investigate the pinout, how is it powered, how many digital IOs should be used to connect it to FPGA, also IO voltage. In future hardware revision I could make some changes for main board, that will help NV2 owners to bring alive this sensor too.
 

Online Hydron

  • Frequent Contributor
  • **
  • Posts: 473
  • Country: gb
I agree that a visual camera in the MSX style (i.e. for edge enhancement etc) isn't necessary with a high resolution (>240p) thermal imager. If you're using the camera for security or something however it can be useful as a separate imager, e.g. to zoom in on someone that you've detected with a wider angle thermal imager.

I'm very impressed with this project - I know a little (mostly theoretical) about thermal imager design and it's a significant undertaking, normally involving a commercial team and significant money! Very interested to learn more.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
I'm very impressed with this project - I know a little (mostly theoretical) about thermal imager design and it's a significant undertaking, normally involving a commercial team and significant money! Very interested to learn more.
Thanks!

I agree that a visual camera in the MSX style (i.e. for edge enhancement etc) isn't necessary with a high resolution (>240p) thermal imager. If you're using the camera for security or something however it can be useful as a separate imager, e.g. to zoom in on someone that you've detected with a wider angle thermal imager.
Integration of visual camera is quite challenging, though not impossible. The problem is that right now there is no free IOs on headers that connect boards with each other, while IO muxing is also not easy. I also doubt that, for example, in UAV mode this visual camera will be better that specialized FPV one. Though at the same time you could easily multiplex outputs of FPV visual and thermal camera. Generally, I think there should be really strong arguments for visual camera integration.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Main board (M-board):

[attach=1] [attach=4]

The brain of the whole device is a Spartan-6 FPGA. As a memory buffer I decided to use HyperRAM. HyperRam memory controller was also developed from the scratch, as there was no high performance implementation of this IP-core in the opensource space. In fact this IP-core is my separate project.
You could ask me, why didn't I use free FPGA's integrated memory controller with DDR2/DDR3 memory. The thing is a very high power consumption, DDR3 + integrated controller would burn even much power, than the whole camera consums itself.

The red bond wire is a bug fix. Though is not the last bug. Using Spartan-6 was a mistake. This FPGA is quite powerful, but also old, has a very poor support, no updates, some bugs in IDE, endless loops in peripherial drivers... :palm:
That's why I'm planning to replace it with Spartan-7 XC7S50 in the same package. I hope the migration will be easy, as previously I had some experience with Xilinx 7-series FPGAs.

The whole stack:
[attach=2] [attach=3]
« Last Edit: July 17, 2020, 01:49:39 pm by VGN »
 

Online Hydron

  • Frequent Contributor
  • **
  • Posts: 473
  • Country: gb
The brain of the whole device is a Spartan-6 FPGA. As a memory buffer I decided to use HyperRAM. HyperRam memory controller was also developed from the scratch, as there was no high performance implementation of this IP-core in the opensource space. In fact this IP-core is my separate project.
You could ask me, why didn't I use free FPGA's integrated memory controller with DDR2/DDR3 memory. The thing is a very high power consumption, DDR3 + integrated controller would burn even much power, than the whole camera consums itself.
Commercial camera cores use things like PSRAM (psuedo-static RAM) for this reason also - lower power consumption than DDR2/3 plus controller.
 

Offline conmega

  • Contributor
  • Posts: 22
  • Country: us
The red bond wire is a bug fix. Though is not the last bug. Using Spartan-6 was a mistake. This FPGA is quite powerful, but also old, has a very poor support, no updates, some bugs in IDE, endless loops in peripherial drivers... :palm:
That's why I'm planning to replace it with Spartan-7 XC7S50 in the same package. I hope the migration will be easy, as previously I had some experience with Xilinx 7-series FPGAs.

It should be pretty simple yes... Stuff like clocking/serdes units have minor changes and such but I have done a bit of similar porting in the past. Even from Altera parts to Xilinx parts from tmbinc's Altera based FPDLink receiver over to a Xilinx 7 series part. It was only a few days of banging my head against the wall. Mostly because I had to learn migen :) Otherwise just some primitives that should need to be changed and some logic wrapped around possibly.
 

Offline ArsenioDev

  • Regular Contributor
  • *
  • Posts: 56
  • Country: us
Holy crap this is basically the HOLY GRAIL for my projects. I'll be watching this with INTENSE interest, it's essentially my dreams come true.
 

Offline Miek

  • Contributor
  • Posts: 26
  • Country: gb
Since you're planning an FPGA swap anyway, it's worth considering the Lattice ECP5 series - the open-source toolchain for them is very good these days. Also, it would probably tempt more people to get involved once you open source the project.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Holy crap this is basically the HOLY GRAIL for my projects. I'll be watching this with INTENSE interest, it's essentially my dreams come true.
Thanks! All the fun is ahead  ;)

Since you're planning an FPGA swap anyway, it's worth considering the Lattice ECP5 series - the open-source toolchain for them is very good these days.
I was thinking about that long ago, when realized that swap is needed. Well, yes, open-source toolchain is very cool and important (especially for security based projects), but I'm not completely sure, that it is worth doing that. I have several arguments:

1. I've already started migration to S7. I'm also afraid that migration from Xilinx to Lattice will be not easy, nor fast. Whole design is AXI4/AXI4-Stream based. I use Xilinx DMA, SERDES... Generally, Lattice swap will cause too massive rework.

2. Lattice ECP5 40nm is a pretty good power optimized solution, but power consumption is not a problem even with Spartan-6 45nm. I hope Spartan-7 28nm will burn even fewer amount of power.

3. I have almost no experience with Lattice, so I'm a bit confused with ECP5 IO performance. ECP5 has 4 SERDES channels, while Xilinx has SERDES on each IO. Will it be enough for HyperRAM and HDMI simultaneous work? BTW, not also sure that ECP5 supports TMDS standard (for HDMI). Is it possible to stream HDMI directly from FPGA without a PHY? If yes, why do they use here HDMI PHYs with ECP5 for such a simple project?
http://www.latticesemi.com/-/media/LatticeSemi/Documents/UserManuals/1D2/FPGA-UG-02036-A.ashx?document_id=52240

Meanwhile Spartan-7 with 1080p 60Hz and even 1440p 30Hz output:
http://labs.domipheus.com/blog/hdmi-over-pmod-using-the-arty-spartan-7-fpga-board/

4. If we try to compare capacity somehow: LFE5UM-45 < XC7S50 < LFE5UM-85
LFE5UM-45 should be excluded, as it looks like to be equal to XC6SLX45, that I use now, but keep in mind that ECP5 is LUT4, while S6 and S7 are LUT6.
Obviosly, we should choose between XC7S50 and LFE5UM-85.

Digikey prices:
LFE5UM-85F-7BG381C ~38$
XC7S50-1CSGA324C    ~53$

Lattice price is lower, but not as much, also FPGA's price is not a big part of the total device cost.

Сorrect me if I'm wrong.


Also, it would probably tempt more people to get involved once you open source the project.
It will be open source anyway, please, be patient  :)
Let's resolve hardware issues first. I hope next week I will commit preliminary schematics (rev.C) for community review and discussion.
« Last Edit: July 21, 2020, 01:23:49 am by VGN »
 

Offline firehopper

  • Frequent Contributor
  • **
  • Posts: 386
  • Country: us
also would be nice for those uninitiated, what to look for to get these modules. the nv3 ones.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
also would be nice for those uninitiated, what to look for to get these modules. the nv3 ones.
66549322653 or 9322653 - BMW
A2229052805 - Mercedes
4G0980552/4G0980552A - AUDI/WV?
 
The following users thanked this post: Fraser, BravoV, Hydron, ArsenioDev, ajw107

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
The focus system:

[attach=11]

There are four SLA printed parts:

[attach=1][attach=4][attach=5][attach=6][attach=7]

1. wheel
2. worm
3. motor mount
4. worm mount

Focus board:

[attach=2][attach=3]

One of the most tricky things was the connection of focus wheel with the lens holding ring. You cannot imagine how much I hate glue, that's why you didn't see any glue connection in the whole design. But...unfortunatelly there were no other cheap and easy way to connect this two parts. Small drops of cyanoacrylate on each tip of the focus wheel allows to reliably connect this two parts so, that I cannot break it by hand. I also tried epoxy, but the adhesion was very poor, so cyanoacrylate is the best for gluing this type of plastic. At the same time the whole focus mechanism is quite lightweight and there is no need to apply much force to make the lens move.

[attach=9][attach=10][attach=8]

The lens holding ring thread allow to make 5 turns of the focus wheel, until it fully unscrews and falls out, that's why we can use 4 turns to adjust the focus. The focus distance changes non-linearly with the wheel position, and that is very good, because a single turn from the fully closed position allows to cover focus distances from ~30cm to infinity, while for macro you will need to make few other turns.

You can also see a photomicrosensor that allows to precisely control the lens position. It is mostly needed to keep the lens away from endpoints, preventing lens lock and unscrew. Also this sensor will be helpful for autofocus! The wheel has 25 teeth, the worm has a single thread and 4 blades, that cross the photomicrosensor slot, so if we detect both rising and falling edges of the pulse, we will have 25x4x2 = 200 points of lens position per a single turn, i.e. accuracy of 1,8 degree.

The focus board thickness is 3mm, it also has 2 additional support points (two 10mm white threaded spacers), that allows to keep sufficient construction rigidity.
The worm mount has a special brass sleeve, that helps to minimize the wear.

The focus motor is a 6mm dia coreless motor with 3 stage built in planetary gearbox (1:136). It is very powerful and consums few energy, but motor vibration is high and rigid connection to the PCB causes some sound amplification. But I have a solution for this, the motor mount is to be redesigned, I'm planning to add a special silicone cover for the motor that will damp the vibrations.

Two words about focus gears reliability. In fact I do not see any wear of the worm or wheel, because after proper UV curing this type of plastic becomes very strong. But I plan to make a continues stress test to define the possible wear level. Making this complex parts on SLA printer is quite cheap and at the same time  quality and reliability is high, that's why I'm planning to use this solution in release design. I have a small experienсe with SLA printing, that's why feel free to point out flaws.

Some common view:

[attach=12][attach=13][attach=14][attach=15][attach=16][attach=17][attach=18]

Of course, all plastic parts STEP models will be avaliable.



P.S. no video yet, but maybe you can enjoy animation :)

[attachimg=19]
« Last Edit: July 22, 2020, 03:46:50 pm by VGN »
 
The following users thanked this post: Fraser, lukier

Offline Ismsanmar

  • Contributor
  • Posts: 21
  • Country: es
Perhaps the vibrations come from the lack of lubrication of the gears, apart from the slightly rough surface finish. You should apply some type of non-hygroscopic based grease, such as PTFE grease.
 

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 9543
  • Country: gb
This is the most impressive thermal camera project that I have seen to date. You have manufactured a very professional mechanical and electronic solution. If I was working on your PCB’s I would honestly think that they were the product of one of the well known thermal camera manufacturers  :-+ Most impressive  and I can only dream of having such design skills  :-+ :-+

Fraser
« Last Edit: July 22, 2020, 09:22:10 pm by Fraser »
 
The following users thanked this post: VGN

Offline Vipitis

  • Frequent Contributor
  • **
  • Posts: 675
  • Country: de
  • aspiring thermal photography enthusiast
That is so beautifully engineered, my plans for a 3D printed follow focus (external) that I sketched out during some downtime at an exam look pathetic.

I will take a deep look at this completely project and see which parts I can somehow use for my project. Experience and tools is what I feel I am lacking.
 
The following users thanked this post: VGN

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Perhaps the vibrations come from the lack of lubrication of the gears, apart from the slightly rough surface finish. You should apply some type of non-hygroscopic based grease, such as PTFE grease.
When I disassembled this motor, I found out that there was no any grease at all in the gearbox. I tried to apply a bike grease (the only grease I had), but things got worse, the "no load" current increased. Looks like too much grease in the gearbox prevented normal rotation, amount of lubrication must be proper. On the other hand, by default at max voltage (3.0V) the shaft rotates at 240RPM, so with ratio 1:136 we have motor rotating at about 32640 RPM, this is a lot. I suppose that probably vibration is caused by rotor disbalance.

This is the most impressive thermal camera project that I have seen to date. You have manufactured a very professional mechanical and electronic solution. If I was working on your PCB’s I would honestly think that they were the product of one of the well known thermal camera manufacturers  :-+ Most impressive  and I can only dream of having such design skills  :-+ :-+
Many thanks for your support! :)

That is so beautifully engineered, my plans for a 3D printed follow focus (external) that I sketched out during some downtime at an exam look pathetic.
Thank you! Just keep up. If your design or piece of a code starts looking pathetic for you, that means your skills are growing, this is a good sign. :-+ Wish you luck with your project! ;)

I will take a deep look at this completely project and see which parts I can somehow use for my project. Experience and tools is what I feel I am lacking.
Feel free to use any piece of this design. I'm going to start commiting some sources by the next week. I'll leave a link to my github repo.
 
The following users thanked this post: zrq

Offline dnhkng

  • Contributor
  • Posts: 11
  • Country: de
Soooo, VGN is the initials of three people, right? Vince, Gary, and Nancy? Because this project seems to be the work of a team of hardware designers, FPGA engineers, and electronics engineers all working together  :-+

Really impressive work, bravo!
 

Offline GeorgeOfTheJungle

  • Super Contributor
  • ***
  • Posts: 2689
  • Country: tr
 

Offline Flanbix

  • Contributor
  • Posts: 30
  • Country: gb
  • If you don't know, ask. If you know, share.
This project is great, lots of features. It is looking quite professional !  :-+
Keep the image and details comming.
 

Offline ajw107

  • Newbie
  • Posts: 4
  • Country: gb
Just wanted to say I joined this forum just to say thank you, and how impressed I am with your design.  I can;t wait until we can order the components.  Just got myself a new Audi 4GO980552A camera off eBay for £295 so I can do this project.  Should be buy the FPGA now as well, or will you be selling all the circuit boards as a kit?
If any one is interested in getting the same camera, there is one left at that price (less than a half the normal price for a used one) and the link is:
https://www.ebay.co.uk/itm/293490373204
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Soooo, VGN is the initials of three people, right? Vince, Gary, and Nancy? Because this project seems to be the work of a team of hardware designers, FPGA engineers, and electronics engineers all working together  :-+
Really impressive work, bravo!
Thank you! Hahaha, not Gary, me and Nancy call him Gabe ;)

This project is great, lots of features. It is looking quite professional !  :-+
Keep the image and details comming.
Thanks!

Just wanted to say I joined this forum just to say thank you, and how impressed I am with your design.  I can;t wait until we can order the components.  Just got myself a new Audi 4GO980552A camera off eBay for £295 so I can do this project.  Should be buy the FPGA now as well, or will you be selling all the circuit boards as a kit?
If any one is interested in getting the same camera, there is one left at that price (less than a half the normal price for a used one) and the link is:
https://www.ebay.co.uk/itm/293490373204
Thank you! I'm really very glad, that you and other people are interested in this project, that additionally motivates to continue. Yes, I have plans to make a some kind of kit, that anyone can buy. This kit will include everything you need, except the thermal camera, sd card, battery (due to some problems with air transportation of batteries). Later I will determine the exact kit contents. There is no need to buy a FPGA for this project. The design of this camera is developed in a way that you will not need even a solder iron to put all parts together. I will also make efforts to keep the functionality and price competetive with other brands, though that will be not easy.
« Last Edit: July 26, 2020, 09:14:57 am by VGN »
 

Offline zrq

  • Contributor
  • Posts: 25
  • Country: cn
This project is one of the most exciting projects I seen on this forum, I never expected a personal project can be so complete and elegant.
I'm considering buying a second hand BMW camera for the FPA, I wonder is there any pitfalls to avoid?
I would be also quite interested in the kit (given the price is reasonable  ;) ), as I don't have the equipment and experience to work on BGA packages.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
This project is one of the most exciting projects I seen on this forum, I never expected a personal project can be so complete and elegant.
I'm considering buying a second hand BMW camera for the FPA, I wonder is there any pitfalls to avoid?
I would be also quite interested in the kit (given the price is reasonable  ;) ), as I don't have the equipment and experience to work on BGA packages.
Thank you!  :)
Concerning pitfalls, I have described some of them in my previous post:
https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/msg3134494/#msg3134494
Actually, if the front window is not cracked and no one disassembled the camera, it should be in a good condition. If the window is cracked, the probability of sensor and lens survival is still quite high, but of course not 100%. As you can see two of my cameras with cracked windows have a good condition of lens and the sensor is working too. It's mostly a matter of price and luck. You can buy a new one for 300-500$ or take a chance with <100$ camera with cracked window. I bought two with cracked windows only because the price was low, I didn't expect that they will be working.

Concerning the kit, I believe you will not need to have any extra tools or skills to build and even calibrate the camera yourself.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf