OpenIRV. ISC0901B0 (Autoliv NV3, FLIR E4/5/6/8) based opensource thermal camera

[VGN]

Hello everyone!

This topic is about my attempt to develop an opensource high performance and multifunctional thermal camera based on a ISC0901 sensor.

I use thermal imaging to diagnostic malfunctions of different devices under development or repair. Five years ago I have developed a small camera based on FLIR Lepton 2 80x60. It works pretty well, but in fact this is a toy for me, and I was looking for something better. Unfortunately FLIR EX series didn't suit for me too for many reasons, including poor fixed focus lens, a bit inconvenient form factor, price, etc...
I'm also a supporter of opensource projects, but all of them are still based on FLIR Lepton 2/3 products or some other tiny sensors.


Once an Avtoliv NV3 camera module occured in my hands. I've disassembled it and found out that this camera module has a great potential and perfectly suits my new project, because:
1. It is based on very popular ISC0901 336x256 sensor. Popular, because NV3 camera is used on tons of different vehicles till now, you can find this spare part very easily. FLIR EX and ETS320 are based on the same sensor.
2. It is an external shutter based camera. External shutter allows to significially improve the image quality.
3. It has a pretty fast (15mm in dia) telescopic lens! This lens is much better that FLIR EX lens.
4. Pretty nice form factor (60x62mm)


I decided to develop a camera that will be multifuntional and can be used in many different ways and applications:
1. Hand tool, like FLIR EX
2. Stand tool, like ETS320
3. Thermal camera for robotics and UAVs, like FLIR Vue


Supported outputs:
1. 320x240 TFT LCD
2. USB UVC (a common webcam protocol)
3. HDMI 480p/720p (probably even 1080p with Spartan-7 FPGA)
4. AV (NTSC/PAL and many other formats if necessary)
5. Micro SD card (SPI mode + SDIO x4 in future)
6. 3-pin configurable 5V tolerant GPIO to support any low speed interface (UART, SPI, I2C, 1-Wire, PWM, PPM, SBUS...)


Power:
1. Standard Li-ion 18350 battery. Battery life is about 3-3.5 hours (with 1100ma battery), depends on number of active outputs, LCD backlight, etc.
(I'm still working on increasing this value, but I think that reaching 4 hours is a good result.)
2. USB charge (<500ma). USB power is enough for continues camera run with all outputs enabled.
3. External power over aux connector 5.0-35.0V (i.e. up to 8S battery) for UAV mode.


Some features:
1. Adjustable motorized focus system (details later).
2. Supports frame rate up to 60fps (details later).
3. Two standard 1/4" metal mounts on the top and bottom.

More photos/videos and detailed stories of each side of this device will be later too.


Repository:

Follow this project on GitHub: https://github.com/OVGN/OpenIRV

Some sources are not available right now, but I have plans to publish them a bit later.


For beta-testers, developers and backers:

Consider to email me if you would like to be a beta-tester, developer or just a backer.
It would be great if you keep mail header like this: "OpenIRV.Developer/Tester" or "OpenIRV.Backer" or etc...
I promis not to publish your email anywhere else and do not send mails, that are not relevant to OpenIRV project.
I always reply letters. You can easily find my email in attached images

[Cat]

Awesome, this looks very promising
I'm looking forward to learn more about your project, especially how you interfaced the µbolometer and implemented UVC.
Didn't think it would be possible to get a motorized focus inside the tiny housing.
Keep up the good work!

[Fraser]

What a superb project, well executed and very professional in its appearance    

As I own one of these cameras I will await further details with GREAT interest 

Fraser

[zrq]

Quite interested in the interface with the sensor and the reverse engineering. Can't wait to see the code! (and hopefully writeup)

[VGN]

Awesome, this looks very promising
I'm looking forward to learn more about your project, especially how you interfaced the µbolometer and implemented UVC.
Didn't think it would be possible to get a motorized focus inside the tiny housing.
Keep up the good work!

Thank you! Looking ahead, I used FX2LP for USB. FPGA is streaming data over 8-bit bus. As a start point I used this Cypress project: https://community.cypress.com/docs/DOC-14406. More details further... With regard to the focus system, first I didn't believe too)

What a superb project, well executed and very professional in its appearance    
As I own one of these cameras I will await further details with GREAT interest 

Wow, many thanks!   As you are owner of one of these cameras, I hope for your help with testing

Quite interested in the interface with the sensor and the reverse engineering. Can't wait to see the code! (and hopefully writeup)

Hi! The interface with the sensor is, obviously, the most tricky moment. In fact, there are a lot of things that I still don't understand, but I hope for community help.

[conmega]

I also have an NV3 and was working on reverse engineering efforts to interface with the original circuitry.
I just got some boards I made for interfacing to the NV2 going out to some friends to help with development/testing. Although its mostly used as a test platform not such a professional looking project as yours. I look forward to your progress and wish you luck!

[tmbinc]

Very cool project!

Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?

Can't wait to see more details!

[VGN]

I also have an NV3 and was working on reverse engineering efforts to interface with the original circuitry.
I just got some boards I made for interfacing to the NV2 going out to some friends to help with development/testing. Although its mostly used as a test platform not such a professional looking project as yours. I look forward to your progress and wish you luck!

Hi! I think we could colaborate for better result. I don't have NV2, as I know, this is a previous generation camera, it has an internal shutter (behind the lens). Sure you know, there is a great work about NV2: https://debugmo.de/2018/12/autoliv-nv2-teardown/

Very cool project!
Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?
Can't wait to see more details!

Hello! Thanks much! Yes, you are right. I refused using original circuitry in my project, because it may not survive, when the front window is broken. Also original electronics limits some opportunities.

[Max Planck]

Bought some time ago an NV3, but never had time to work on it.

Thus, I'm really impressed by the project.
Are you using just the FPA or also some parts of the NV3?

Max
 

[VGN]

Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?

Thus, I'm really impressed by the project.
Are you using just the FPA or also some parts of the NV3?

Thanks you! Ok, I thinks this will be the start point of the story.

I have three cameras, two of which have cracked front windows (btw cost me under 100$ for each). I didn't expect that condition of the most critical parts, i.e. sensor and lens will be perfect. In fact all three sensors are alive with amount of bad pixes <1%.



As you know, the front window is very fragile. As soon as it catches a stone from the road and cracks, your camera will soon stop working, because of water. Some water damage pictures:


Nevertheless the sensor itself in located in the center of the camera module, and incomming water do not reach it, as the water level do not rise above the bottom edge of the broken window. Perhaps that's why both sensors of drowned cameras are working. Of course you should be carefull, there is no 100% garanthy that sensor will work, but at the same time the probability is quite high. The shutter motor can also be easily replaced if necessary.

You can also see a lot of small dents at the surface of the windows. But the surface of all lens are in perfect condition. I think this is because lens is located very deep inside of the enclosure and also protected by broken window. Anyway that gives a chance to bring alive very poor cameras.


Here you can see parts that I have reused from the original camera:


The front window and cap is optional. I don't really use them, because it is an additional obstacle for IR radiation and the device itself is not water or dust protected. It is designed mostly for home and limited outdoor usage.

[Fraser]

So if it is just the casing, lens, FFC flag and Microbolomter PCB that is used in your design, presumably this project could be adapted to the earlier NV2 model ?

You have basically built a thermal imaging camera from scratch which is a significant achievement when you do not have the datasheet for the microbolometer or its recommended bias voltage values. Building the back end video processing package and firmware is a further very impressive achievement. You have skills  Much respect for you 

Fraser

[VGN]

So if it is just the casing, lens, FFC flag and Microbolomter PCB that is used in your design, presumably this project could be adapted to the earlier NV2 model ?

Yes, you are right. I believe it can be adapted to any camera, that needs deep raw image processing.

You have basically built a thermal imaging camera from scratch which is a significant achievement when you do not have the datasheet for the microbolometer or its recommended bias voltage values. Building the back end video processing package and firmware is a further very impressive achievement. You have skills  Much respect for you 

Thanks much! Bias values were picked by hand, I'm still looking for a way to calculate this values properly. Do you have any knowleges about that? After tons of experiments I found out that this is 6-bit value per pixel and could properly feed them to FPA core.

[Fraser]

The bias values are normally provided by the fabricator of the microbolometer and are unique to each one. The user of the microbolometer then uses these values as a start point from which they are fine tuned for the cameras particular application, temperature range etc. The bias voltages can be fine tuned for best NeTD. Sadly I do not have any bias voltage values for FLIR microbolometers.

Fraser

[Treehouseman]

I've been thinking that it could be possible to extract the bias values from the flash chips, I would imagine it's just a semi standardized mapping that if you compare 2 camera's flash images and see what the differences is it may stand out. It would then be a task to see how they're storing it but with the NV2 given that we have the code from it's softcore, the softcore may load in the values, and that could tell us how they're stored. Especially if they keep the same structure they used for debugging where you could also send it values manually for initial calibration.

While we don't currently have the program from the nv3 it's possible it uses the same data encoding. On top of that we could also see how it's used in a Flir E4 or a Flir E40 (NV3 and NV2 respectively) and see if it's the same thing. I would assume they just drop the same table of values in.

[VGN]

Well, there are a lot of things to tell about, so for simplicity I'm going to move step by step from outside to inside.

The whole enclosure consists of four parts:
1. Top (where the display is mounted)
2. Middle
3. Bottom
4. Original black casing

All parts were developed for CNC machining, but I decided to try SLA 3D printing for prototype. Of course, the quality is not super, there are some wracks at the surface and edges. Anyway I'm sure CNC machining with further casting plastic in silicone molds will give much better quality.



The bottom:



The middle:



The metal mounts is nothing but a 1/4" to 3/8" thread adapter. Fitting the adapter from the inside prevents possiblity of unwanted unscrew:



The top:



Finally I had to increase the lens hole a little to prevent field of view narrowing, because previously I had to raise this black case up to place the focus mechanics.
There is a round boarder that helps to make a neat cutout. This cutout will still allow you to fit the front window if you need it, but maybe you will also have to increase the hole of the window cap too.

[Fraser]

Great to read this thread as it gradually reveals the work you have done. Loving this 

Fraser

[Max Planck]

First technical question.
I know, it is maybe to early at this stage, but have you taken into account the heating of your camera enclosure during operation (electronics, battery, external heat sources) and its temperature changes on camera (FPA) operation?

Max

[VGN]

Great to read this thread as it gradually reveals the work you have done. Loving this 

Thanks a lot!

First technical question.
I know, it is maybe to early at this stage, but have you taken into account the heating of your camera enclosure during operation (electronics, battery, external heat sources) and its temperature changes on camera (FPA) operation?

Yes, it's taken into account. As you know, all FPA pixel are different, they all have different response to the same amount of IR radiation. So to solve this problem and make the image uniform we should determine a gain and an offset parameter for each pixel. Gain parameter changes with the camera temperature quite slowly, whereas the pixel offset changes at a temperature difference of ~0.2°C. To compensate this temperature drift FFC (Flat-field correction) procedure is initiated. We use external shutter to close the front of the lens and assuming that shutter surface is uniform, we perform recalculation of offset parameter, improving the image. You can run FFC manually any time you want to improve the image quality or enable automatic mode, when firmware monitors enclosure temperature by integrated temperature ICs and initiates FFC periodically.

[VGN]

The display module.



This is the biggest LCD that I could find to fit in 60x62mm dimentions. Also manufacturer promises long term supply for this part (10 years since 2018).

Brass thread inserts are glued into plastic by cyanoacrylate. That looks ugly, that's why plastic molding is preferable for manufacturing.

Buttons cover is made of platinum cured silicone with black pigment. It's pretty reliable and looks good. As you see the button's plunger is quite small, so I decided to place additional 0.3mm plastic plunger that will allow to increase button's press area. Plastic plungers were cutted by hand and looks also ugly, but laser cut will fix that.

There is also a small bug, the buttons board should have additional mounting hole in the center. Will be fixed in future revision.

Few words about electronics. The CLC filter reduces power noise from AUX port. DC-DC converts 5.0V-35.0V to 4.5V. Unfortunately, there was no enough space to fit this regulator on the mother board, that's why it is here. All inductors are shielded to reduce possible EMI noise. LCD backlight is powered by constant current regulation IC (not PWM) also to minimize the power noise level.




Feel free to ask questions and criticize the design, that may help to make it better.

[newex]

The amount of work you have done is amazing.
I think it makes no sense to criticize the design, because we do not see the schematic diagram, but it looks great and professional.

[Vipitis]

I am more than impressed by the effort so far. Wish I had the knowledge, experience and tools to mount such a project. But really shows what can be done... and motivates me to move on with my project (in 14 days.)

[Fraser]

I am loving this thread. I very much look forward to seeing future updates. Your camera project looks very professional and not unlike what I have seen in R&D prototyping. Very professional in all respects.

I might add that even with the skills to create the mechanical and electronic design, that is not enough for such a project, as I well know. The fact that you also have the skills required to write the code for the hardware is very impressive

Fraser

[Max Planck]

Yes, it's taken into account. As you know, all FPA pixel are different, they all have different response to the same amount of IR radiation. So to solve this problem and make the image uniform we should determine a gain and an offset parameter for each pixel. Gain parameter changes with the camera temperature quite slowly, whereas the pixel offset changes at a temperature difference of ~0.2°C. To compensate this temperature drift FFC (Flat-field correction) procedure is initiated. We use external shutter to close the front of the lens and assuming that shutter surface is uniform, we perform recalculation of offset parameter, improving the image. You can run FFC manually any time you want to improve the image quality or enable automatic mode, when firmware monitors enclosure temperature by integrated temperature ICs and initiates FFC periodically.

Yes, you have two things, NUC and thermal dirft caused by several factors. I was asking about the latter. You are correcting it using a shutter. My question was about the dynamics of that process, i.e. how often the shutter has to be activated to keep a decent image quality/temperature measurement accuracy.

Knowing about some of the challenges you were and are facing, I'm impressed.

Max 

[VGN]

The amount of work you have done is amazing.
I think it makes no sense to criticize the design, because we do not see the schematic diagram, but it looks great and professional.

Thank you! Well, yes, I realize that this hard to criticize the design without the schematics, but I mean, maybe you have some global ideas that do not require schemes currently. The problem is that I'm going to make quite a lot of changes for the next hardware revision and I think that it would be better if I publish the very last schematics. I believe we will have enough time to discuss and make changes to design if necessary before I send new files for manufacturing.

I am more than impressed by the effort so far. Wish I had the knowledge, experience and tools to mount such a project. But really shows what can be done... and motivates me to move on with my project (in 14 days.)

Thanks much! I wish you good luck with your project!

I am loving this thread.
...

Many thanks, Fraser!

Yes, you have two things, NUC and thermal dirft caused by several factors. I was asking about the latter. You are correcting it using a shutter. My question was about the dynamics of that process, i.e. how often the shutter has to be activated to keep a decent image quality/temperature measurement accuracy.
Knowing about some of the challenges you were and are facing, I'm impressed.

Thank you, Max! Basically one should perform FFC every time when camera's temperature changes for a some threshold value up or down. Also there is no need to perform a FFC if temperature is stable, the image quality will not get worse in time. I can't exactly say how often the shutter has to be activated, as it depends on the environment temperature, how fast the camera heats up and the temperature change threshold value. I don't think that this a point of worry, until you use it on UAV, because FFC causes an image freeze for a short period of time (~1 sec). There are two ways to mitigate this problem. First - you can initiate FFC yourself, second - the firmware can display a countdown before FFC to help the pilot get ready.

[VGN]

Peripheral board (P-board):



Some descriptions:



This board do not process image at all, but provides output path to different interfaces, manages battery power, powerpath logic and some other stuff. D-board (display) and B-board (buttons) are connected to it through 12-pin and 5-pin JST connectors. You can also see a 6-pin JTAG header that is used for FPGA debug on the M-board (main), which I will show futher.



Some device safety features:

• All IOs at HDMI, USB and 7-pin AUX port are ESD protected.
• USB 5V power line has overcurrent (>500ma) and overvoltage (up to 20V) protection.
• AV output has an output buffer with short-to-ground and short-to-battery (up to 18V) protection.
• AUX external power line has high side reverse polarity protection. (high side protection type is probably not the best choice...)
• Three GPIOs at the 7-pin AUX port that goes directly to FPGA are protected by 5V tolerant bidirectional buffers.
• Battery reverse polarity protection. Magic smoke will not appear even if you install battery wrong for a continues period of time.
• Battery is also protected by a special charge/discharge voltage/current protection IC like in factory protected Li-ion batteries. I refused using protected Li-ions, as their length can vary too much from manufacturer to manufacturer, you simply won't be able to insert the battery into holder, meanwhile dimentions of unprotected batteries are much more predictable.

Other features:

• RTC provides timestamps for SD card filesystem.
• Temperature sensor.
• Power reset button.

[joe-c]

its a really awesome project.
I am very interested.

currently, I work on an open-source solution too, but its more an interface of an existing Camera, not a completely new device.

I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.

anyway... really nice. Thanks for show it.
best regards

[Ismsanmar]

VGN, incredible work. Another one here interested in the possibility of adapt your solution for the NV2 ISC0601 sensor. I'm another one with one of those lying around.

I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.

The problem with that camera is that only goes at 15fps on line detection mode

[VGN]

its a really awesome project.
currently, I work on an open-source solution too, but its more an interface of an existing Camera, not a completely new device.
I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.

Thank you!
No there is no visual camera, but I thought about that. You know, I held Flir EX in my hands once. And I can't say that MSX technology impressed me a lot. MSX cannot help when FPA's resolution is low itself or when you cannot change the lens focus. Also image alignment is not possible at short distances. Looks like MSX brings more engineering headache than benefits. Are there a lot of people using MSX? Maybe you have some arguments that can change my mind?

VGN, incredible work. Another one here interested in the possibility of adapt your solution for the NV2 ISC0601 sensor. I'm another one with one of those lying around.

Thanks!
Yes, I think it's possible to adapt this camera for NV2 sensor. But I don't have it, and know completely nothig about ISC0601 sensor. If you have this camera, you could investigate the pinout, how is it powered, how many digital IOs should be used to connect it to FPGA, also IO voltage. In future hardware revision I could make some changes for main board, that will help NV2 owners to bring alive this sensor too.

[Hydron]

I agree that a visual camera in the MSX style (i.e. for edge enhancement etc) isn't necessary with a high resolution (>240p) thermal imager. If you're using the camera for security or something however it can be useful as a separate imager, e.g. to zoom in on someone that you've detected with a wider angle thermal imager.

I'm very impressed with this project - I know a little (mostly theoretical) about thermal imager design and it's a significant undertaking, normally involving a commercial team and significant money! Very interested to learn more.

[VGN]

I'm very impressed with this project - I know a little (mostly theoretical) about thermal imager design and it's a significant undertaking, normally involving a commercial team and significant money! Very interested to learn more.

Thanks!

I agree that a visual camera in the MSX style (i.e. for edge enhancement etc) isn't necessary with a high resolution (>240p) thermal imager. If you're using the camera for security or something however it can be useful as a separate imager, e.g. to zoom in on someone that you've detected with a wider angle thermal imager.

Integration of visual camera is quite challenging, though not impossible. The problem is that right now there is no free IOs on headers that connect boards with each other, while IO muxing is also not easy. I also doubt that, for example, in UAV mode this visual camera will be better that specialized FPV one. Though at the same time you could easily multiplex outputs of FPV visual and thermal camera. Generally, I think there should be really strong arguments for visual camera integration.

[VGN]

Main board (M-board):



The brain of the whole device is a Spartan-6 FPGA. As a memory buffer I decided to use HyperRAM. HyperRam memory controller was also developed from the scratch, as there was no high performance implementation of this IP-core in the opensource space. In fact this IP-core is my separate project.
You could ask me, why didn't I use free FPGA's integrated memory controller with DDR2/DDR3 memory. The thing is a very high power consumption, DDR3 + integrated controller would burn even much power, than the whole camera consums itself.

The red bond wire is a bug fix. Though is not the last bug. Using Spartan-6 was a mistake. This FPGA is quite powerful, but also old, has a very poor support, no updates, some bugs in IDE, endless loops in peripherial drivers...
That's why I'm planning to replace it with Spartan-7 XC7S50 in the same package. I hope the migration will be easy, as previously I had some experience with Xilinx 7-series FPGAs.

The whole stack:

[Hydron]

The brain of the whole device is a Spartan-6 FPGA. As a memory buffer I decided to use HyperRAM. HyperRam memory controller was also developed from the scratch, as there was no high performance implementation of this IP-core in the opensource space. In fact this IP-core is my separate project.
You could ask me, why didn't I use free FPGA's integrated memory controller with DDR2/DDR3 memory. The thing is a very high power consumption, DDR3 + integrated controller would burn even much power, than the whole camera consums itself.

Commercial camera cores use things like PSRAM (psuedo-static RAM) for this reason also - lower power consumption than DDR2/3 plus controller.

[conmega]

The red bond wire is a bug fix. Though is not the last bug. Using Spartan-6 was a mistake. This FPGA is quite powerful, but also old, has a very poor support, no updates, some bugs in IDE, endless loops in peripherial drivers...
That's why I'm planning to replace it with Spartan-7 XC7S50 in the same package. I hope the migration will be easy, as previously I had some experience with Xilinx 7-series FPGAs.

It should be pretty simple yes... Stuff like clocking/serdes units have minor changes and such but I have done a bit of similar porting in the past. Even from Altera parts to Xilinx parts from tmbinc's Altera based FPDLink receiver over to a Xilinx 7 series part. It was only a few days of banging my head against the wall. Mostly because I had to learn migen Otherwise just some primitives that should need to be changed and some logic wrapped around possibly.

[ArsenioDev]

Holy crap this is basically the HOLY GRAIL for my projects. I'll be watching this with INTENSE interest, it's essentially my dreams come true.

[Miek]

Since you're planning an FPGA swap anyway, it's worth considering the Lattice ECP5 series - the open-source toolchain for them is very good these days. Also, it would probably tempt more people to get involved once you open source the project.

[VGN]

Holy crap this is basically the HOLY GRAIL for my projects. I'll be watching this with INTENSE interest, it's essentially my dreams come true.

Thanks! All the fun is ahead 

Since you're planning an FPGA swap anyway, it's worth considering the Lattice ECP5 series - the open-source toolchain for them is very good these days.

I was thinking about that long ago, when realized that swap is needed. Well, yes, open-source toolchain is very cool and important (especially for security based projects), but I'm not completely sure, that it is worth doing that. I have several arguments:

1. I've already started migration to S7. I'm also afraid that migration from Xilinx to Lattice will be not easy, nor fast. Whole design is AXI4/AXI4-Stream based. I use Xilinx DMA, SERDES... Generally, Lattice swap will cause too massive rework.

2. Lattice ECP5 40nm is a pretty good power optimized solution, but power consumption is not a problem even with Spartan-6 45nm. I hope Spartan-7 28nm will burn even fewer amount of power.

3. I have almost no experience with Lattice, so I'm a bit confused with ECP5 IO performance. ECP5 has 4 SERDES channels, while Xilinx has SERDES on each IO. Will it be enough for HyperRAM and HDMI simultaneous work? BTW, not also sure that ECP5 supports TMDS standard (for HDMI). Is it possible to stream HDMI directly from FPGA without a PHY? If yes, why do they use here HDMI PHYs with ECP5 for such a simple project?
http://www.latticesemi.com/-/media/LatticeSemi/Documents/UserManuals/1D2/FPGA-UG-02036-A.ashx?document_id=52240

Meanwhile Spartan-7 with 1080p 60Hz and even 1440p 30Hz output:
http://labs.domipheus.com/blog/hdmi-over-pmod-using-the-arty-spartan-7-fpga-board/

4. If we try to compare capacity somehow: LFE5UM-45 < XC7S50 < LFE5UM-85
LFE5UM-45 should be excluded, as it looks like to be equal to XC6SLX45, that I use now, but keep in mind that ECP5 is LUT4, while S6 and S7 are LUT6.
Obviosly, we should choose between XC7S50 and LFE5UM-85.

Digikey prices:
LFE5UM-85F-7BG381C ~38$
XC7S50-1CSGA324C    ~53$

Lattice price is lower, but not as much, also FPGA's price is not a big part of the total device cost.

Сorrect me if I'm wrong.

Also, it would probably tempt more people to get involved once you open source the project.

It will be open source anyway, please, be patient 
Let's resolve hardware issues first. I hope next week I will commit preliminary schematics (rev.C) for community review and discussion.

[firehopper]

also would be nice for those uninitiated, what to look for to get these modules. the nv3 ones.

[VGN]

also would be nice for those uninitiated, what to look for to get these modules. the nv3 ones.

66549322653 or 9322653 - BMW
A2229052805 - Mercedes
4G0980552/4G0980552A - AUDI/WV?

[VGN]

The focus system:



There are four SLA printed parts:



1. wheel
2. worm
3. motor mount
4. worm mount

Focus board:



One of the most tricky things was the connection of focus wheel with the lens holding ring. You cannot imagine how much I hate glue, that's why you didn't see any glue connection in the whole design. But...unfortunatelly there were no other cheap and easy way to connect this two parts. Small drops of cyanoacrylate on each tip of the focus wheel allows to reliably connect this two parts so, that I cannot break it by hand. I also tried epoxy, but the adhesion was very poor, so cyanoacrylate is the best for gluing this type of plastic. At the same time the whole focus mechanism is quite lightweight and there is no need to apply much force to make the lens move.



The lens holding ring thread allow to make 5 turns of the focus wheel, until it fully unscrews and falls out, that's why we can use 4 turns to adjust the focus. The focus distance changes non-linearly with the wheel position, and that is very good, because a single turn from the fully closed position allows to cover focus distances from ~30cm to infinity, while for macro you will need to make few other turns.

You can also see a photomicrosensor that allows to precisely control the lens position. It is mostly needed to keep the lens away from endpoints, preventing lens lock and unscrew. Also this sensor will be helpful for autofocus! The wheel has 25 teeth, the worm has a single thread and 4 blades, that cross the photomicrosensor slot, so if we detect both rising and falling edges of the pulse, we will have 25x4x2 = 200 points of lens position per a single turn, i.e. accuracy of 1,8 degree.

The focus board thickness is 3mm, it also has 2 additional support points (two 10mm white threaded spacers), that allows to keep sufficient construction rigidity.
The worm mount has a special brass sleeve, that helps to minimize the wear.

The focus motor is a 6mm dia coreless motor with 3 stage built in planetary gearbox (1:136). It is very powerful and consums few energy, but motor vibration is high and rigid connection to the PCB causes some sound amplification. But I have a solution for this, the motor mount is to be redesigned, I'm planning to add a special silicone cover for the motor that will damp the vibrations.

Two words about focus gears reliability. In fact I do not see any wear of the worm or wheel, because after proper UV curing this type of plastic becomes very strong. But I plan to make a continues stress test to define the possible wear level. Making this complex parts on SLA printer is quite cheap and at the same time  quality and reliability is high, that's why I'm planning to use this solution in release design. I have a small experienсe with SLA printing, that's why feel free to point out flaws.

Some common view:



Of course, all plastic parts STEP models will be avaliable.


P.S. no video yet, but maybe you can enjoy animation

[Ismsanmar]

Perhaps the vibrations come from the lack of lubrication of the gears, apart from the slightly rough surface finish. You should apply some type of non-hygroscopic based grease, such as PTFE grease.

[Fraser]

This is the most impressive thermal camera project that I have seen to date. You have manufactured a very professional mechanical and electronic solution. If I was working on your PCB’s I would honestly think that they were the product of one of the well known thermal camera manufacturers  Most impressive  and I can only dream of having such design skills 

Fraser

[Vipitis]

That is so beautifully engineered, my plans for a 3D printed follow focus (external) that I sketched out during some downtime at an exam look pathetic.

I will take a deep look at this completely project and see which parts I can somehow use for my project. Experience and tools is what I feel I am lacking.

[VGN]

Perhaps the vibrations come from the lack of lubrication of the gears, apart from the slightly rough surface finish. You should apply some type of non-hygroscopic based grease, such as PTFE grease.

When I disassembled this motor, I found out that there was no any grease at all in the gearbox. I tried to apply a bike grease (the only grease I had), but things got worse, the "no load" current increased. Looks like too much grease in the gearbox prevented normal rotation, amount of lubrication must be proper. On the other hand, by default at max voltage (3.0V) the shaft rotates at 240RPM, so with ratio 1:136 we have motor rotating at about 32640 RPM, this is a lot. I suppose that probably vibration is caused by rotor disbalance.

This is the most impressive thermal camera project that I have seen to date. You have manufactured a very professional mechanical and electronic solution. If I was working on your PCB’s I would honestly think that they were the product of one of the well known thermal camera manufacturers  Most impressive  and I can only dream of having such design skills 

Many thanks for your support!

That is so beautifully engineered, my plans for a 3D printed follow focus (external) that I sketched out during some downtime at an exam look pathetic.

Thank you! Just keep up. If your design or piece of a code starts looking pathetic for you, that means your skills are growing, this is a good sign. Wish you luck with your project!

I will take a deep look at this completely project and see which parts I can somehow use for my project. Experience and tools is what I feel I am lacking.

Feel free to use any piece of this design. I'm going to start commiting some sources by the next week. I'll leave a link to my github repo.

[dnhkng]

Soooo, VGN is the initials of three people, right? Vince, Gary, and Nancy? Because this project seems to be the work of a team of hardware designers, FPGA engineers, and electronics engineers all working together 

Really impressive work, bravo!

[GeorgeOfTheJungle]

 

[Flanbix]

This project is great, lots of features. It is looking quite professional ! 
Keep the image and details comming.

[ajw107]

Just wanted to say I joined this forum just to say thank you, and how impressed I am with your design.  I can;t wait until we can order the components.  Just got myself a new Audi 4GO980552A camera off eBay for £295 so I can do this project.  Should be buy the FPGA now as well, or will you be selling all the circuit boards as a kit?
If any one is interested in getting the same camera, there is one left at that price (less than a half the normal price for a used one) and the link is:
https://www.ebay.co.uk/itm/293490373204

[VGN]

Soooo, VGN is the initials of three people, right? Vince, Gary, and Nancy? Because this project seems to be the work of a team of hardware designers, FPGA engineers, and electronics engineers all working together 
Really impressive work, bravo!

Thank you! Hahaha, not Gary, me and Nancy call him Gabe

This project is great, lots of features. It is looking quite professional ! 
Keep the image and details comming.

Thanks!

Just wanted to say I joined this forum just to say thank you, and how impressed I am with your design.  I can;t wait until we can order the components.  Just got myself a new Audi 4GO980552A camera off eBay for £295 so I can do this project.  Should be buy the FPGA now as well, or will you be selling all the circuit boards as a kit?
If any one is interested in getting the same camera, there is one left at that price (less than a half the normal price for a used one) and the link is:
https://www.ebay.co.uk/itm/293490373204

Thank you! I'm really very glad, that you and other people are interested in this project, that additionally motivates to continue. Yes, I have plans to make a some kind of kit, that anyone can buy. This kit will include everything you need, except the thermal camera, sd card, battery (due to some problems with air transportation of batteries). Later I will determine the exact kit contents. There is no need to buy a FPGA for this project. The design of this camera is developed in a way that you will not need even a solder iron to put all parts together. I will also make efforts to keep the functionality and price competetive with other brands, though that will be not easy.

[zrq]

This project is one of the most exciting projects I seen on this forum, I never expected a personal project can be so complete and elegant.
I'm considering buying a second hand BMW camera for the FPA, I wonder is there any pitfalls to avoid?
I would be also quite interested in the kit (given the price is reasonable  ), as I don't have the equipment and experience to work on BGA packages.

[VGN]

This project is one of the most exciting projects I seen on this forum, I never expected a personal project can be so complete and elegant.
I'm considering buying a second hand BMW camera for the FPA, I wonder is there any pitfalls to avoid?
I would be also quite interested in the kit (given the price is reasonable  ), as I don't have the equipment and experience to work on BGA packages.

Thank you! 
Concerning pitfalls, I have described some of them in my previous post:
https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/msg3134494/#msg3134494
Actually, if the front window is not cracked and no one disassembled the camera, it should be in a good condition. If the window is cracked, the probability of sensor and lens survival is still quite high, but of course not 100%. As you can see two of my cameras with cracked windows have a good condition of lens and the sensor is working too. It's mostly a matter of price and luck. You can buy a new one for 300-500$ or take a chance with <100$ camera with cracked window. I bought two with cracked windows only because the price was low, I didn't expect that they will be working.

Concerning the kit, I believe you will not need to have any extra tools or skills to build and even calibrate the camera yourself.

[VGN]

Here some more images, captured right by camera. Original images were in .bmp format, as this is the easiest way for image capturing, but I had to convert it to .png, as eevblog forum do not support .bmp attaches. Of course in future I plan to add png support right on camera, as bmps are much larger.

P.S. videos are comming soon!

[Fraser]

You cannot beat a decent microbolometer coupled with a good lens 

At least you get a decent, low noise, image to process Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

Fraser

[ajw107]

Thank you! I'm really very glad, that you and other people are interested in this project, that additionally motivates to continue. Yes, I have plans to make a some kind of kit, that anyone can buy. This kit will include everything you need, except the thermal camera, sd card, battery (due to some problems with air transportation of batteries). Later I will determine the exact kit contents. There is no need to buy a FPGA for this project. The design of this camera is developed in a way that you will not need even a solder iron to put all parts together. I will also make efforts to keep the functionality and price competetive with other brands, though that will be not easy.

That is great news, as I'm no good with a soldering iron (rubbish eyesight), although i will keep on trying to give it a go, much to the deprement of my fingers.  I have to admit, I have also just gone and bought a new SLA printer (Kelant S500, bit of a gamble as no reviews - but on paper is sounds good for the price), as my old ANET8 has long since given up the ghost and I've been itching to try an SLA for years.  The photos are great by the way, is that a brain cross-section in photos 11-14? I know it can't be, but it looks like it, he he!  Looking forward to being able to order the curcuitrs when they become available, so excited, like a kid with a new toy.

[ajw107]

Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).

[VGN]

Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).

This thread is actually the guide.
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...

[VGN]

I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...

You cannot beat a decent microbolometer coupled with a good lens 
At least you get a decent, low noise, image to process Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

The lens definitly matters. I don't have Flir E8 right now, but I remembered that earlier I have made a some photos at home. Here is the same bathroom with OpenIRV and FLIR E8 (hacked E4). Of course, shooting time and some other conditions are different:

[VGN]

I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...


Earlier I showed some images, shooted in good conditions, when the scene tempretature difference is high. As you could see, there is no noticeable noise.
Now, let's look at abilities of this camera in poor conditions.

First, so that you can better understand what's going on, here is a photo of bicycle at the wall with a cup of warm water. The room temperature is +25.5 °С , the water in cup is +29.5 °С (measured by external thermometer). Temperature difference of the coldest and hottest objects is about 4-5°С. As you can see, the image is quite good, further we will analyze this scene without a cup of warm water.




The same scene without a cup of warm water. This gif is a sequence of images shooted with a small time gap in a raw mode (no any kind of filtering):



As you can see, the noise gets out, as temperature range is probably about ~1°С. The noise pattern is complex and quite specific. After reading tons of literature and some analysis and I came to think that there are two main noise components:
1. Row noise. This is a temporal noise, i.e. random fluctuations of the whole pixel line, that randomly changes every moment.
2. Column noise. This is a FPN (fixed pattern noise) that changes quite slowly and cannot be fully removed even by FFC (flat field correction) procedure.


Earlier I have developed a special averaging module that helps to average a sequence of multiple frames comming out of the sensor. Max possible number of averaged frames is 262144, though reasonable values are 2, 4, 8, 16, 32, 64, 128. Signal averaging is a common technique, intended to increase the strength of a signal relative to noise. So, let's try to use this module with different averaging values:



We can see that starting from x8 value the row noise disapears, that confirms the nature of row noise. At the same time column noise is still present, averaging do not help to neutralize this type of noise, we need something more.


Few days ago I found a way to remove this column FPN in frequency domain by FFT (fast fourier transform) and a custom filter. As a test scene I decided to use some uniform object. Here you can see how it works. Image from camera (uniform body) is on the left, FFT of this image is on the right. Black lines on the FFT plot are the actual filter function, that "cuts out" the noise components. Here is the link to the online FFT tool that I used for experiments: https://www.ejectamenta.com/Fourifier-fullscreen/




At last I decided to test both averaging and FFT filters at the complex scene with a lot of details:



Also keep in mind that this experiments are done on equalized 8-bit image. I think that final image quality will be a little better if I apply FFT filter to the raw 14-bit pixel frames.


I don't want to start developing this FFT filter right now, it will be easier to develop it for new hardware and new FPGA. Though averaging is a good filter, it reduces the frame rate, that's why I should implement something else for row noise, of maybe FFT will be enogh, the time will show. At least I know how to deal with this noise. If you have good ideas and suggestions for image filtering, please don't keep in secret


P.S. This is how lepton 80x60 "sees" this bicycle: 

[zrq]

For the similar noise I noticed on Xtherm cameras, I wrote a routine to reduce the single column noise by filtering in the temporal space.

Code: [Select]

merit = np.abs(cv2.Laplacian(cv2.GaussianBlur(imgFlt, (3, 3), 3), cv2.CV_32F)) / 65536
merit = 1 / pow(merit + 0.005, 3)
leftErr = imgFlt[:, 0:leftHalf] - cv2.GaussianBlur(imgFlt[:, 0:leftHalf], (1, 5), 10)
horiLeftAcc = np.average(leftErr, weights=merit[:, 0:leftHalf], axis=1)
rightErr = imgFlt[:, leftHalf:] - cv2.GaussianBlur(imgFlt[:, leftHalf:], (1, 5), 10)
horiRightAcc = np.average(rightErr, weights=merit[:, leftHalf:], axis=1)
vertErr = imgFlt - cv2.GaussianBlur(img, (5, 1), 10)
vertAcc = np.average(vertErr, weights=merit, axis=0)
for n in range(0, np.size(img, 0)):
    imgFlt[n, 0:leftHalf] = cv2.subtract(imgFlt[n, 0:leftHalf], (k[0] * horiLeftAcc[n])).squeeze(1)
    imgFlt[n, leftHalf:] = cv2.subtract(imgFlt[n, leftHalf:], (k[1] * horiRightAcc[n])).squeeze(1)
for n in range(0, np.size(img, 1)):
    imgFlt[:, n] = cv2.subtract(imgFlt[:, n], (k[2] * vertAcc[n])).squeeze(1)
However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked.

[VGN]

However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked.

I made the same mistake at first)
I also think that parametrizing this filter pattern like this will help to find optimal configuration:




By the way, I found a cool Protoshop plug in for FFT noise reduction.
http://ft.rognemedia.no/

I tested it on bicycle image, drawing that cross filter pattern by hand. Now no image crop, no averaging, only pure FFT filter.

Before:



After:



You can see some luminance fluctuation in filtered image, I'm sure that this is caused by changes in histogram equalization, caused by noise peaks. That's why we should apply this filter on 14-bit raw image before histogram gain correction. Anyway, there is almost no high frequency row/column noise and that is really good!

[ajw107]

This thread is actually the guide.
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...

You know, I thought that as soon as I pressed the post button (that's what I get for asking questions when I'm half asleep).  In my head there was some magical M12 CAN -> USB connector, he he!  Any news on when things will be ready for us to try (no rush, just over excitment, like a child with a new toy)!

[Hydron]

These noise issues are present on other microbolometers too - e.g. the ULIS 17u 640x480 FPA has a well known issue with column noise, and there are some filtering techniques that have been tried to improve it (some suggested by ULIS themselves I think). I'm not sure whether I have any info about these, but if I do and they are able to be distributed (unfortunately unlikely) I'll do so.

[VGN]

Hello!

Sorry for the delay, I was working at the next hardware revision. I have a lot of progress and updated. But first, the video capture, that I have promised long ago:



I doubt that youtube will ever show this video in original quality. So, for those of you, who would like to see the original quality and frame rate, here is the link to my google drive (video file is about 180MB): https://drive.google.com/file/d/1QfK6TBKxTnJjb9-Vjo1R4MsX56F_mhFq/view?usp=sharing
Google drive will suggest you to watch it online in browser, but you should better download it.

Video was captured over USB. Note that this thermal camera do not record the sound. The sound was added later.

[Jenny]

Hi, I’m so amazed (like others also do) at how incredible work you have done. 
Just a quick question regarding to the last video: I realized you have motorized focus on the camera? Is it auto or manual? I found it is a bit slow, but stop at the focal point without overshoot. Is it phase detection or even laser assisted?
Again, it’s really unbelievable how you done such work, in a short time...

[VGN]

Hi, I’m so amazed (like others also do) at how incredible work you have done. 
Again, it’s really unbelievable how you done such work, in a short time...

Thank you! I'd not say, I have done it in a short time..

Just a quick question regarding to the last video: I realized you have motorized focus on the camera? Is it auto or manual? I found it is a bit slow, but stop at the focal point without overshoot. Is it phase detection or even laser assisted?

Yes, the focus is motorized, you can find focus system PCB photos in my previous posts. There are two modes, manual and auto. On the video, you saw a manual mode, I changed the focus myself. Autofocus fuction is under development now. It is based on image sharpness detection. Right now it is implemented in software and works too slow and not precize. I definitly should make a hardware acceleration of image sharpness calculation. In this case I will be able to detect sharpness gradient change at the highest frame rate, and accidental sharpness change due to the hand shake will not confuse the algorithm, I hope.

Regarding to the speed, yes it is slow now, because I have limited the RPM. The focus motor is rated for 3.0V, but it is running at ~2.0V now. Both shutter and focus motors are powered almost directly from Li-ion battery, which voltage is variable (4.2V-3.0V) and a little higher than motor recommended voltage. As I don't want to overvoltage the motor power, I'm using PWM to limit the voltage. Further I'm going to adjust the PWM duty cycle according to the battery voltage to keep it at 3.0V.

There is also another feature. We have 4 lens turns to adjust the focus. The focus distance changes non-linearly with the lens position, and that is very good, because a single turn from the fully closed position allows to cover focus distances from ~30cm to infinity. Lens movement under single turn will be pretty quick, thought for macro mode you will have to make few other turns.

Forgot to say, there is one more way to increase the focusing speed by using motor with smaller gear ration. Right now reductor ration is 1:136. But there same type of motors with 1:26 gearbox, with higher RPM. If torque will be enough, it will be worth replacing the motor.

[ajw107]

I have to admit, I was getting a little worried when we hadn't heard from you in a while, so pleased it is going well.  The video quality is incredible, it seems to have come on leaps and bounds from the still very good bike photos we had earlier.  Wonder if I could ask a few questions:
1)  Will the FPGA be able to handle the thermal data processing and the auto focus and keep the frame rate?  I'm not familiar with the Spartan-6 board?
2) Is there an ETA on when we can get a look at some of the code and stl files?  I'm just being impatient with excitement, I know.  But The code will be fascinating to look through for the image processing alone, and the stl files will be a good test for my printer (whenever it eventually arrives, grrr).
3) I'm not entirely sure where abouts you are in your project roadmap, it seems you are getting close to a finished product, just requiring some some tweaks (sometimes the phase that takes the longest time, but hopefully everyone on here can help once it goes on github).
4) Any idea of how much the kit minus the nv3 will cost?  I'm moving house at the moment, so counting the pennies, and want to make sure I put some money aside.  Even better if you could say some thing like the circuitry will cost about X and the housing/gears/internal parts should be about Y, etc.

Anyway, as I say, magnificent work, it really is impressive what you have accomplished.

[VGN]

I have to admit, I was getting a little worried when we hadn't heard from you in a while, so pleased it is going well.

Sorry for this, I will try not to make long delays or at least warn about that.

1)  Will the FPGA be able to handle the thermal data processing and the auto focus and keep the frame rate?  I'm not familiar with the Spartan-6 board?

New hardware will be based on newest Spartan-7. Of course, FPGA will be able to handle the thermal data processing, autofocus and many other things simultaneously. FPGA is very good with paraller processing.

2) Is there an ETA on when we can get a look at some of the code and stl files?  I'm just being impatient with excitement, I know.  But The code will be fascinating to look through for the image processing alone, and the stl files will be a good test for my printer (whenever it eventually arrives, grrr).
3) I'm not entirely sure where abouts you are in your project roadmap, it seems you are getting close to a finished product, just requiring some some tweaks (sometimes the phase that takes the longest time, but hopefully everyone on here can help once it goes on github).

Don't get me wrong, this is my first open source project. Though I have developed quite a lot of different devices in a small teams with private repositories, I don't have an experience with open source and wide community. I'm not sure I know the proper way of organizing such kind of projects. Is it better to make a public repo from the very beginning or keep it private for those of you, who would like to collaborate and reveal sources with the first batch of devices? I've always thought about it. Maybe you or anybody else can give an advice?

Also I didn't expect that I will make so many changed for the new revision, this is another reason that I haven't published schematics yet... . Tomorrow I will tell about most of new features.

Regarding to the technical side of the project, in general terms, this is my plan for nearest future:
1. Finish new hardware revision schematics (will be finished in a week).
2. Purchase components and new PCBs (shipping is about 2-3 weeks).
3. Rewrite the HyperRAM memory controller for Spartan-7 and publish it as a separate project.
4. By this time I hope to get new PCBs and component and can assembly everything.
5. Start porting the design to new hardware.

If everything is ok at this step and I'm sure that hardware is working properly, I could make a first small batch of devices for developers and testers. After that I think we will be ready for larger production. This is a matter of discussion.

4) Any idea of how much the kit minus the nv3 will cost?  I'm moving house at the moment, so counting the pennies, and want to make sure I put some money aside.  Even better if you could say some thing like the circuitry will cost about X and the housing/gears/internal parts should be about Y, etc.

I will provide this information as soon as I get the final BOM. But I'm going to keep the price competitive to HT-301, TE-Q1, HT A2, FLIR C2 and other "cheap" thermals, though OpenIRV is going to be more powerful and multifuctional. Also keep in mind that the batch size makes difference on the final price.

[railrun]

If you are looking for tester I would be happy to be a tester.

[bicycleguy]

@VGN
Looks very interesting.

Do you know if a 4H0980552 is an acceptable version.  Note the H instead of G.  How would I find out what that means?

thanks

[VGN]

If you are looking for tester I would be happy to be a tester.

I'm very glad, thank you! That will be cool if you and all other people, who would like to be a tester or even a developer of the first kit batch, email me (view my profile for email).

Do you know if a 4H0980552 is an acceptable version.  Note the H instead of G.  How would I find out what that means?

I don't know what actually each letter means, but according to the photos, 4H0980552 is the previos generation (NV2). This project initially was designed for NV3, but I have plans to make hardware in a way to support more types of sensors.

[tmbinc]

The easiest take-away is the connector - it's always this HSD ("weird automotive never seen before") connector, and either there are two additional pins next to it (NV2) or not (NV3). NV2 has Power (separate), CAN and LVDS so it needs 6 pins, whereas NV3 has Power and this "weird automotive never seen before bidirectional highspeed interface that's neither CAN nor LVDS but can replace both", and hence only needs 4 pins.

The case is also distinctively different, but harder to describe/see.

[VGN]

The easiest take-away is the connector - it's always this HSD ("weird automotive never seen before") connector, and either there are two additional pins next to it (NV2) or not (NV3). NV2 has Power (separate), CAN and LVDS so it needs 6 pins, whereas NV3 has Power and this "weird automotive never seen before bidirectional highspeed interface that's neither CAN nor LVDS but can replace both", and hence only needs 4 pins.
The case is also distinctively different, but harder to describe/see.

Exactly!

[bicycleguy]

Thanks for the help.  Had a seller on eBay tell me they were the same.  He said the 4H was from an Audi A8 instead of A7.  Luckily I saw your quick responses and hadn't paid yet. 

thanks.

[tmbinc]

Well for NV2 we have https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/, but VGN's work of course is far more impressive.

[VGN]

Well for NV2 we have https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/, but VGN's work of course is far more impressive.

tmbinc, btw I have to notice that I saw your investigation of NV2 on debugmo.de long ago. That is an incredible work that inspired me to work on NV3! I'm a newbie on this forum, I haven't remembered your nick-name, so I only recently realized that was you, haha)) 
Long story short, very glad to meet you!

[VGN]

I'm finishing new hardware revision and want to tell you about main changes:

1. Most of the changes are refered to M-board (main). I have replaced old Spartan-6 XC6SLX45 with new Spartan-7 XC7S50.

2. I decided to add more RAM. I found new HyperRAM part which is twice cheaper and +20% faster than the previous one. The reason of adding more RAM is that probably image filtering with FFT will require more memory bandwidth and I'm afraid that single channel RAM will not be enough. Anyway it is always better to have a footprint for an additional memory rank, than do not have it at all.

3. From some point I realized that it is worth to support or, better say, give a chance to support more types of sensors than only NV3's one.

NV3's sensor ISC0901B0 is connected to the main board over 40-pin connector. There are:

• 1 pin is connected to sensor frame (chassis)
• 2 pins for bias power (11.0v)
• 2 pins for core power (3.3v)
• 2 pins for io power (2.5v)
• 6-pins to control the sensor. clk, ena, cmd, bias, data_odd, data_even. Details further.
• 2 pins are NC (not connected to sensor die and anywhere else)
• 25 pins (remaining) are ground

Well, leaving 25 pins for ground is ridiculous, that's why I have developed a new backward compatible pinout that will use some ground pins as IOs and connected them to FPGA. If this extended IOs will be configured as input at FPGA side, the direct path to ground over ISC0901B0 board will not stress FPGA's bank.

Generally now we have 18 single ended IOs that are connected to a single FPGA bank. The bank power can be configured in safe range (0.6v - 3.3v) over DC-DC's feedback resistor divider. Other board curcuits have separate power, that's why IO voltage change will not affect anything else. I hope these amount of IOs will be enough to connect any sensor. Probably it is worth to route some of this IOs in differential line manner, but I'm not sure. Any ideas or suggestions are welcome.

Also two NC pins I have connected to system power (battery rail) as an additional power path.
Bias and core power rails are also configurable or can be disabled if desired.

4. I have reworked whole power system, removed 2 LDOs and improved efficiency of some power rails.

5. You can also see some strange 10-pin blue connector. There are 7 IOs (3.3V fixed), ground and two power rails (regulated 3.3V + battery rail). It can be used for any general purpose, but I have one quite interesting task for this connector, details will be furter.


Regarding to the support of different sensors. We have a base part and any other sensor can be connected over special adapter board. Optionally we will also need a cover to protect the sensor. The most common sensors that I think is possible to support, except NV3's are NV2 sensor (ISC0601B), probably Seek Thermal Mosaic Cores and of course Lepton2/3 (though this hardware for such a tiny core is an overhead...)

[tmbinc]

VGN, thanks! I'm deeply impressed by your reverse-engineering, electrical design, industrial design _and_ project presentation skills. I can't wait to use your design and play around with it!

I think there are great benefits of this project that go far above "getting a cheap thermal camera" (which is a very great goal in itself!). It will bootstrap open source efforts to operate on raw thermal sensor data and I see a realistic chance of building an image processing pipeline that matches (and exceeds) that of commercial products. The math is not trivial, but also not super hard, and it is about time that we don't rely on companies that tend to intentionally cripple image quality (noise generator, mandatory fusion and cropping - WTF) on the affordable range of thermal imagers, and take this back into our hands.

[VGN]

Possible configuration without the focus board. In this case we should change the bottom enclosure part. It has the same outline, but different height (13mm to 7mm).

[VGN]

Welcome to my GitHub repo!   

https://github.com/OVGN/OpenIRV

All mechanical parts STEP models are avaliable. You can try to print some of them for a test. That will be cool if you show your attempts to us, especially gears. I have used SLA printing, but didn't try any other technologies. Keep in mind that this parts are not release version, I'm going to make some changes, even not backward compatible.

[ajw107]

That is wonderful news, it's quite late here now but just got my SLA today, so will definitely have a play when I get round to assembling it later this week.

Welcome to my GitHub repo!   

https://github.com/OVGN/OpenIRV

All mechanical parts STEP models are avaliable. You can try to print some of them for a test. That will be cool if you show your attempts to us, especially gears. I have used SLA printing, but didn't try any other technologies. Keep in mind that this parts are not release version, I'm going to make some changes, even not backward compatible.

[VGN]

Thanks to Treehouseman, looks like even forum users could not email me, that's strange.

So, here is my email both for registered and non-registered users. I hope this "light" obfuscation will protect me from spam somehow 
Consider to email me if you would like to be a beta tester, developer or just a backer.
It would be great if you keep mail header like this: "OpenIRV.Developer/Tester" or "OpenIRV.Backer" or etc...

I promis not to publish your email anywhere else and do not send mails, that are not relevant to OpenIRV project.

[bicycleguy]

I just received an e-bay NV3 that seems in near new condition with just a plastic tab on the window holder broken.  Opening the case reveals a ribbon cable from below going up to the window that is not shown in your pictures.  What is that for?  I can probably get a picture, but its pretty tight.  How do I unhook it?

[VGN]

I just received an e-bay NV3 that seems in near new condition with just a plastic tab on the window holder broken.

This window holder locking plastic tab breaks very easily. Mine was broken too)

Opening the case reveals a ribbon cable from below going up to the window that is not shown in your pictures.  What is that for?  I can probably get a picture, but its pretty tight.  How do I unhook it?

This is actually an internal heater with integrated SMD thermistor. Original camera activates this heater in cold weather conditions to melt the ice at the front window, as ice is not IR transparent. There is a FPC connector at the other end of this cable. The heater itself is glued to the plastic housing by double-sided tape. You don't need this part with OpenIRV.

[Treehouseman]

I just received an e-bay NV3 that seems in near new condition with just a plastic tab on the window holder broken. 

So you're the one that bought that camera! I went to go grab it the other day because 5% ebay bucks and it was gone. Ended up going for the Mercedes one instead.

[bicycleguy]

Here's an attempt to print the worm with a regular printer, Prusa Mk3s.  Looks bad but in my experience  it will probably still work.
" alt="" class="bbc_img" />

[ArsenioDev]

Damn you lot snapping up all the good deals on them, back to monitoring the alerts smh

[VGN]

Here's an attempt to print the worm with a regular printer, Prusa Mk3s.  Looks bad but in my experience  it will probably still work.

Nice try!) I counted 31 layer per 5.1mm or looks like your layer height is around 0,15mm. I think you probably can get much better quality with FDM if you make layer height 0,05mm, exact this layer height I used while printing my parts on SLA.

[bicycleguy]

Nice try!) I counted 31 layer per 5.1mm or looks like your layer height is around 0,15mm. I think you probably can get much better quality with FDM if you make layer height 0,05mm, exact this layer height I used while printing my parts on SLA.

Yes on both.  Never tried smaller than 0,15.  I have a 0,3 nozzle to try too.
Here's an uncleaned-up gear for test.


Just curious, is it necessary to use black?

[VGN]

Yes on both.  Never tried smaller than 0,15.  I have a 0,3 nozzle to try to.  Here's an uncleaned-up gear for test.

Yes, this one looks much better! But I'm not sure about that four tips. Anyway, FDM looks promissing.

Just curious, is it necessary to use black?

Do you mean plastic color? What's the difference?


BTW, I haven't still tell about the lens internals. If you unscrew the lens holding ring, you can find a lot of interesting parts. We are going to use only the lens and the lens holding ring (first two positions on photo). Also we need some spring to compensate the thread backlash. This is extreamely needed for proper autofocusing, as backlash causes an error when the motor changes rotation direction. The lens movement distance from end to end is 2mm, and we also have 2mm of space from the bottom of the housing. Generally we need a 4mm height light spring with 2mm working stroke. Unfortunatelly, we cannot reuse original wave spring, as it is too stiff, because it was designed only to lock the holding ring from being unscrewed.

Frankly speaking, that was a great headache, as manucturing a new special spring could be too costy. Manufacturers just hangs up the phone if you are going to purchase less than thousands of spring... But luckily I found a solution - stencils! The stencils, that are usually used to transfer solder paste to a bare PCBs. I decided to design a flat spring, based on this technology. This is going to be reliable and very cheap! Another good feature is that stencils are stainless.

I haven't finished the spring design yet, I should define the number, the width and the length of "legs", the stencil thickness. There are some early examples in attachments. Maybe you noticed a mistake on renders, that legs ends should be bended too.

[bicycleguy]

Just curious, is it necessary to use black?

Do you mean plastic color? What's the difference?
The inside of the gear beveled surface is nearly a lens shade.  On a visible light camera this would all be black.  Is this necessary?  Almost seems like white would emit less IR.  But I'm ignorant of this IR stuff.

I have designed a few stainless steel springs however.  The material is critical.  Not sure about stencil material.  If you can bend it sharply and it doesn't spring back its the wrong kind.

[VGN]

The inside of the gear beveled surface is nearly a lens shade.  On a visible light camera this would all be black.  Is this necessary?  Almost seems like white would emit less IR.  But I'm ignorant of this IR stuff.

The camera FOV is 24° h x 18° v, according to this documentation: http://www.safetyvision.com/sites/safetyvision.com/files/FLIR_PathFindIRII_User_Guide_1.pdf
The angle that this inside of the gear beveled surface forms with main optical axis is wider, than FOV angle of 24° h. In other words, camera do not see any presence of this gear, as it doesn't cross the FOV. That's why I think there is no sense to change the plastic color.

I have designed a few stainless steel springs however.  The material is critical.  Not sure about stencil material.  If you can bend it sharply and it doesn't spring back its the wrong kind.

I believe stencil material and thickness should be stable, as this parameters are very critical for PCB manufacturing. Anyway, this is a matter of experiments.
Looking for good ideas..

[Ruhkukah]

Barev, VGN! Hi all!

I took a stab to print these two challenging focus parts (worm and wheel). Part of the worm broke off after my kid played with it (to speed things up I used 5% infill, so it is pretty fragile)  Now reprinting it
IMO the quality is good enough for the purpose, both parts are very smooth to the touch. For the worm I used black PLA filament, 0.05mm layer height, 0.15mm TriangleLab nozzle and 0.15mm extrusion width, 2 perimeters. For the wheel I tried 0.025mm layer height. My printer is Prusa MK3s stock. I didn't use any supports, so the cross and the circular plate needed a little bit of cleaning.
See attached.

VGN, I applaud your work - it looks very professional and top notch in every aspect. Looking forward to become a tester!

Pavel

[rockwell]

Can we use with this interface the Autoliv NV4 (12µm, 640x480, 50°x39°) sensor in the future ?
Would be cool when these sensor become available and we could use same interface.

[tmbinc]

1. Is the FLIR PathFindIR II _identical_ with the Autoliv NV3? I know it looks pretty much spot on, but I wonder if it's the same thing. Is that an AutoLiv OEM product then? If so, is there also an NV2-based PathFind device?

I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.

2. NV4 - I've heard about it, but haven't seen it. Do you have any details? Is the NV business still at Autoliv or now at Veoneer?

[rockwell]

Veoneer is an Autoliv spin-off.
Unfortunately I don't have more information on the NV4 sensor.

[VGN]

Barev, VGN!

I took a stab to print these two challenging focus parts (worm and wheel). Part of the worm broke off after my kid played with it (to speed things up I used 5% infill, so it is pretty fragile)  Now reprinting it
IMO the quality is good enough for the purpose, both parts are very smooth to the touch. For the worm I used black PLA filament, 0.05mm layer height, 0.15mm TriangleLab nozzle and 0.15mm extrusion width, 2 perimeters. For the wheel I tried 0.025mm layer height. My printer is Prusa MK3s stock. I didn't use any supports, so the cross and the circular plate needed a little bit of cleaning.
See attached.

Barev, Ruhkukah!) Thank you for your test! Looks really good. You'd better try to use 0.025mm layer height for worm too. In this case the worm surface will not be so stepped. Looking forward your results)

VGN, I applaud your work - it looks very professional and top notch in every aspect. Looking forward to become a tester!

Thanks much! Don't forget to email me to become a tester. You can find my email in attached images of the first topic post.

1. Is the FLIR PathFindIR II _identical_ with the Autoliv NV3? I know it looks pretty much spot on, but I wonder if it's the same thing. Is that an AutoLiv OEM product then? If so, is there also an NV2-based PathFind device?

I don't exactly know, but I think, that NV3 is identical to FLIR PathFindIR II with some light changes, that each car manufacturer make to fit this camera in their car.

Can we use with this interface the Autoliv NV4 (12µm, 640x480, 50°x39°) sensor in the future ?
Would be cool when these sensor become available and we could use same interface.

I'm trying to do the best to provide as wide support of different IR cores as it is possible: more RAM, higher RAM throughput, more IOs at the sensor connector, more various adjustable power rails at the same connector, etc. In case of best scenario, we will just have to make a cheep adapter board to connect current hardware with a new sensor and develop a HDL module that will interface with it. Also, probably, a new sensor covering enclosure.

[horstmannhid]

Is it maybe an idea to use a ring magnet instead of the spring for the lens thread compensation? I think for 2mm there is always enough force available.

[VGN]

Is it maybe an idea to use a ring magnet instead of the spring for the lens thread compensation? I think for 2mm there is always enough force available.

You are right, this is definitly a good idea! 

Four 1.5mm height magnets generate enough force, pulling a metal ring, mounted on the lens. I glued magnets directly to the housing, but I think some special thin plastic frame, that will hold magnets properly is strongly needed. We will also have to glue the lens to the focus wheel gear. I used a hot glue gun instead of cyanoacrylate for a test.

Advantages:
> no additional friction!!!
> still cheap and reliable solution

Disadvanteges:
> a bit more complex
> two more gluing points (the lens, the magnet frame)

Have something to think about..

[VGN]

I'm glad to say that PDF schematics of the M-board (the most complex one) and B-board (the most trivial one) are avaliable on my repo!
https://github.com/OVGN/OpenIRV/tree/master/docs/schematics

This is alpha version v0.1.0, I probably will make some changes, but without changelog. First stable version will be v1.0.0.

Looking forward your constructive criticism, bug reports, recommendations, ideas, etc.

In attachments you can see how M-board and B-board looks like now. For M-board I decided to replace JTAG connector, FFC/FPC connector is quite thin and can be easily passed between enclosure parts out of the device for debugging purposes.

I hope other schematics will be finished by the end of this week. I'm going to purchase components for new revision and start porting HyperRAM controller to new FPGA this week. I have a special board from another project that utilized Spartan-7 FPGA and a single HyperRAM IC for testing.
That will be fun)

[horstmannhid]

Four 1.5mm height magnets generate enough force, pulling a metal ring, mounted on the lens. I glued magnets directly to the housing, but I think some special thin plastic frame, that will hold magnets properly is strongly needed. We will also have to glue the lens to the focus wheel gear. I used a hot glue gun instead of cyanoacrylate for a test.

That sounds great for this little test. What I meant is using a ring magnet with the right dimensions. They can be bought via ebay. Have a look at the attached picture. If there is a fitting magnet no additional plastic frame for the magnets is needed. Just glue the ring magnet to the housing.


Had a look at the schematic. Debouncing of the buttons will be done in VHDL via a filter in the FPGA? In the microcontroller unit it is an annoying task to debounce something.

[bicycleguy]

In the original design they seem to have gone to great lengths to keep the lens assembly from rotating when focusing.  There are 4 sliding teflon surfaces between the focusing ring and the lens.  Was that to keep wear particles from the threads off the sensor?

Looks like the foam gasket that seals the bottom of the lens to the housing is not thick enough to seal for close focusing if it requires more than about .5mm expansion from the stock setting. 

A popular way to get around lead screw slop on 3D printers is plastic nuts. They apparently last long enough for that extreme task.  Why not replace the threaded focusing ring with a combined focusing ring + ring gear version.  The SLA printer may be up to the task.  The threads don't have to be perfect, the errors control the slop.  And you get rid of a glue joint !  The threaded portion could be backed with a circlip if required. edit, not enough room

[VGN]

That sounds great for this little test. What I meant is using a ring magnet with the right dimensions. They can be bought via ebay. Have a look at the attached picture. If there is a fitting magnet no additional plastic frame for the magnets is needed. Just glue the ring magnet to the housing.

Yes, a ring magnet would be the best solution. On the other hand, small magnets are very cheap and easy to find, while ring magnets are quite exotic. Anyway I measured mounting seat dimentions, you can find them in attachments.

Had a look at the schematic. Debouncing of the buttons will be done in VHDL via a filter in the FPGA? In the microcontroller unit it is an annoying task to debounce something.

Buttons pull-ups and debouncing capacitors are at the P-board (peripheral). Of course, there is additional debouncing logic in the HDL design. Agree, filtering buttons by internal mcu is a poor idea.

In the original design they seem to have gone to great lengths to keep the lens assembly from rotating when focusing.  There are 4 sliding teflon surfaces between the focusing ring and the lens.  Was that to keep wear particles from the threads off the sensor?

This unit looks quite over engineered for me. Maybe I don't understand something, but really, no idea, why using so many parts just for a fixed focus...

Looks like the foam gasket that seals the bottom of the lens to the housing is not thick enough to seal for close focusing if it requires more than about .5mm expansion from the stock setting. 

Right, but I don't think that we need this foam gasket, it prevents lens rotation a bit. Also this camera is not supposed to be used in any aggressive environment, no water/dust protection. BTW, even Flir Vue Pro is not water/dust resistant.

A popular way to get around lead screw slop on 3D printers is plastic nuts. They apparently last long enough for that extreme task.  Why not replace the threaded focusing ring with a combined focusing ring + ring gear version.  The SLA printer may be up to the task.  The threads don't have to be perfect, the errors control the slop.  And you get rid of a glue joint !  The threaded portion could be backed with a circlip if required. edit, not enough room

I'm afraid, that custom plastic nut probably will rotate too hard, and this tiny focus motor won't be able to make it move. All the more so I'm going to change the focus motor with higher RPM one. We should somehow keep lens rotating very easily and without backlash, otherwise, there is no chance for fast focus adjustment.

[bicycleguy]

Here's what I was thinking of:

There is enough room for a steel ring spring that could be tensioned for a nice fit.  I haven't tried on my filament printer.  Wonder how it would do on SLA.

[VGN]

Here's what I was thinking of:
There is enough room for a steel ring spring that could be tensioned for a nice fit.  I haven't tried on my filament printer.  Wonder how it would do on SLA.

WOW! I take my words back. These cutouts are very interesting! 
That definitly should be tested on SLA, pity that I don't have any 3D printer at all. I could purchase it for test, but only later with other parts, which I'm going to fix a little too.

1. How are you going to make the steal ring?
2. What about a special internal border that will help to center and fit the lens? Check out attachments. But I'm not completely sure how to make it properly, this is just a suggestion.
3. So the holding ring thread pitch is 0.508 (20mil)?! I thought it is metric 0.5mm pitch, though I didn't measure it, not even sure that I could...

[bicycleguy]

1. How are you going to make the steal ring?

Just bend some music wire around a pipe of a guessed dia.

2. What about a special internal border that will help to center and fit the lens? Check out attachments. But I'm not completely sure how to make it properly, this is just a suggestion.

I think the attachment might have messed up, all I see is what I previously posted.    edit  Whoops I didn't look close enough.  Have to think about this a little more.  I was still planning on using a flattened wave spring, but that doesn't make any sense.

3. So the holding ring thread pitch is 0.508 (20mil)?! I thought it is metric 0.5mm pitch, though I didn't measure it, not even sure that I could...

I don't know the pitch either, but the dia. is .998 inch so I assume 1 inch with 50thrds/inch.  Could be something else.  So short not sure it matters, especially with plastic threads.

I thought you had an SLA printer for the parts you posted?

[VGN]

I thought you had an SLA printer for the parts you posted?

No, I don't have it, I just order printing.


P-board schematics are avaliable: https://github.com/OVGN/OpenIRV/tree/master/docs/schematics
Most of the work is done, I just have to finish rounting and check everything very carefully before ordering new PCBs.
Display and Focus boards schematics will be published in a few days.

M-board is also modified a little. Added ESD protection for JTAG. Also increased sensor IOs number from 18 to 20.
I hope that 20 IO + I2C IO-expander is enough to interface most sensors and even high megapix CMOS digital image sensors like MT9F002 and others, for example.

Finally solved the problem with board stacking. Two special M1.6 SMT board spacers allow properly connect main and peripheral boards without original aluminium housing.

[bicycleguy]

@VGN  Thinking about the focus.  Your original design just needs a weaker spring.  The spring deflection is proportional to thickness^3.  Maybe the existing spring can be etched in something like pcb copper etch solution to the proper thickness.  Not sure if it has a coating on it.

[VGN]

@VGN  Thinking about the focus.  Your original design just needs a weaker spring.  The spring deflection is proportional to thickness^3.  Maybe the existing spring can be etched in something like pcb copper etch solution to the proper thickness.  Not sure if it has a coating on it.

I also tend to this. I'm going to try both the stencil spring and the magnet based one.

The spring deflection is proportional to thickness^3.  Maybe the existing spring can be etched in something like pcb copper etch solution to the proper thickness.  Not sure if it has a coating on it.

Not sure if that's going to work. Also far not everyone will be able to repeat this.

[horstmannhid]

Yes, a ring magnet would be the best solution. On the other hand, small magnets are very cheap and easy to find, while ring magnets are quite exotic. Anyway I measured mounting seat dimentions, you can find them in attachments.

You are right. I also didn't find a fitting ring magnet. I think it would be a lot better to use normal magnets.


Had a look at your new schematics. Great work!

For debouncing of the buttons I usually use the following circuit. So the bouncing is filtered even when the button is pushed.



Do you know the standby current of the whole system? Button 3 seems to turn on the whole system.

[VGN]

Had a look at your new schematics. Great work!

Thank you! Feel free to ask me any questions! Any suggestions are welcome for discussing.

For debouncing of the buttons I usually use the following circuit. So the bouncing is filtered even when the button is pushed.

Oh...right, pull-up + RC filter looks much better, going to fix this. Thanks!

Do you know the standby current of the whole system?

Of course, 8uA (micro amps).

Button 3 seems to turn on the whole system.

Right, button 3 and also EXT_GPI/PWR_ENA pin at the X9 connector is used to enable the whole system.

[VGN]

Focusing mechanics with magnet spring. Works pretty well, though a bit noisy, as focus board amplifies the motor vibrations, this is going to be fixed.
I'm also planning to replace the motor with higher RPM one, probably we could achieve faster lens movement and less noise.

https://youtu.be/jcN1_7Xi5mU

[Pantheron]

Dear VGN,

amazing project!
It clearly shows your dedication in every aspect of the complete project.
The lovely designed Focus mechanism.
The Housing.
The Software and even the PCBs are designed with love!

I really appreciate this.


But i also got some questions - sensor related.

You are reading the sensor out with 60fps?
How low can the clock for reading go, so it i spossible to readout at 10fps? or ist just internally fixed to 60fps?

For what is the CMD Pin on the sensor? 1 Wire? Any information on that?




Thank you.

[VGN]

amazing project!
It clearly shows your dedication in every aspect of the complete project.
The lovely designed Focus mechanism.
The Housing.
The Software and even the PCBs are designed with love!

I really appreciate this.

Hi! Many thanks!

You are reading the sensor out with 60fps?

Yes, the raw 14-bit per pixel data is comming out at 60fps. I left in my previos posts a link to my google drive with a video, captured right from ther sensor.
Here is the link: https://drive.google.com/file/d/1QfK6TBKxTnJjb9-Vjo1R4MsX56F_mhFq/view?usp=sharing
I advise you to download (~180MB) this video before viewing, as not smart youtube downgrades video quality too much.

How low can the clock for reading go, so it i spossible to readout at 10fps? or ist just internally fixed to 60fps?

We don't have and I believe will never have access to datasheets of this core, so I actually don't know that. Original board is forwarding clock at 73.636MHz, I'm using the same value. You will be able to experiment with clock rate, as soon as the HDL-design will be published.

On the other hand, I don't think that you really need it. If you would like to integrate this camera with any other hardware, you can choose USB, HDMI, AV or a low speed interface access over GPIO lines at X9 connector of the P-board. I have plans to implement a SPI interface over GPIO line, so that you can control the camera and readout the video data at any FPS you would like.

For what is the CMD Pin on the sensor? 1 Wire? Any information on that?

Forget about any standard interface with this FPA, everything is proprietary, the CMD interface as well. There are a lot of thing to tell about, I'm preparing a special manual for this. Well, the short answer, this is a 1-bit command line, synchronous to the main clock (73.636MHz). It is used to control the sensor. According to my investigations, a special 22 byte command is sent to the sensor to get each new frame. First two bytes of the command are 0x3F and 0xC0. I'm 99.9% sure, that this is a preamble that helps the core to detect a new command. Why? There is no any other strobe to validate the incomming command, 0xC0 = ~0x3F, very likely to be a preamble. Also, the core does not stream anything if I flip any bit in these two bytes. Other 20 bytes keep some special data to control the sensor. Unfortunatelly, the command fields are not byte aligned, that makes investigations a bit harder. There are a lot of bits that do not affect anything. Also a command for one sensor may not fit another, as themal array parameters are very different from core to core. Though I don't know what all the command fields are doing, I could find some common command pattern that should fit all sensors and a single special field that you can quite easily tune to get the image. I have tested this algorithm on my three FPAs and could make all of them working well. I'm sure that the more people will play with this sensor, the faster we will find out the function of each command field. Another way is to reverse engineer the firmware and get this data from original EEPROM. But, I'm not sure that this is a good idea, as this is an intellectual property of Avtoliv/FLIR.

[rockwell]

Do you think it's possible to have the sensor connected via a shielded cable from about 50cms long to the controller electronics?

[VGN]

Do you think it's possible to have the sensor connected via a shielded cable from about 50cms long to the controller electronics?

Do you mean the ISC0901B0 sensor?

[rockwell]

Yes, only the sensor in a compact as possible housing.

[VGN]

Yes, only the sensor in a compact as possible housing.

Sensor lines IO standard is LVCMOS 2V5. Data stream is running at @73.636MHz SDR. 50cm is a quite long distance, I'd not say that this is impossible, but will be a bit challenging. You should take in account proper termination of the I/O lines to exclude signal "ringing". I also think that using buffers/repeatters for the five high speed lines, i.e. clock, bias, data_odd, data_even, cmd is also a good idea.

After that keep in mind that the interface between FPGA and sensor is source synchronous, i.e. FPGA is generating the clock and sends it to the sensor. Bias and cmd line are also synchronous to the rising edge of the outgoing clock. At the same time sensor outputs data_odd and data_even signals synchronous to the falling edge of the incomming clock. Signals over PCB usually travels at a speed of 165ps/inch. So, 50cm gives about 3.2ns of delay. At 73.636MHz clock period is about 13.6ns. In ideal conditions (without your expander) we have 13.6ns/2 = 6,78ns of margin before and after the active edge of the clock signal to properly read incomming data at the FPGA side. But in case of additional delay of 3.2ns, the margin after the active clock edge will be lower and probably we will have to compensate this delay for proper operation. Current FPGA is quite powerfull to make such kind of tricks, so yes, this is possbile in certain conditions.

I'm not sure, but making such a long sensor power lines can also be a problem. The noise level of bias and core power rails should be as low as it is possible. This is a matter of experiments.
If it is not a secret, why do you need such a long cable?

[rockwell]

Many thanks for the comprehensive answer.
And no it's no secret at all, I would like to mount the sensor on a very small pan/tilt head on a Trimble UX5 drone.

[VGN]

Many thanks for the comprehensive answer.
And no it's no secret at all, I would like to mount the sensor on a very small pan/tilt head on a Trimble UX5 drone.

It was worth starting with this)) In this case I don't really think that this expander will be reliable enough. At the same time dimentions and weight of the camera is quite critical for small UAVs, that cannot carry this big body camera. But I think I have a solution.

The M-board, as the most expensive part, mostly because of the FPGA, was designed in a way to be fully independent from all other boards, including P-board. That's why for small UAV applications a new special board should be designed. Check out attachments, I just have copypasted some parts from the other boards.

We don't need focusing board, only shutter. So, right now the camera module that you see in the attachments weights 78g. The new P-board will add another 10g, resulting to 90g.

I have small experience with UAVs, and absolutely no any experience with FPV, FPV-cameras, interfaces, etc... Are you still flying on NTSC or PAL? 
That's why I have some question to you and community:

1. What do you think about this addon? Is it worth doing?
2. What about total weight and dimentions?
3. What kind of interfaces FPV pilots use nowadays? Is AV enough? What is the output format (NTSC/PAL/or...)?
4. What else is reasonable to add to the board?

[rockwell]

It's amazing how you can design and develop in such a short time projects like this !

But in this case, weight is not my problem, my UAV is equipt with a fixed 36 MP mirrorless full frame camera with a 15mm lens and is normally used in mapping and surveying for making orthomosaics in combination with a very capable GPS system.

My problem only is the limited space on the pan-tilt head that I would like to mount on this UAV, after that I removed the above mentioned camera.
Transmission could be done with HDMI or AV, 2.4GHZ or 5.0GHZ FHSS.
I could opt for a bigger pan-tilt had but i'm afraid this causes to much drag for this fixed wing UAV.

[VGN]

Transmission could be done with HDMI or AV, 2.4GHZ or 5.0GHZ FHSS.

Thanks, streaming HDMI is not a problem, though I can't decide what kind of connector is better to use. I saw different FPV HDMI transmitting hardware, everything looks to be proprietary.

I could opt for a bigger pan-tilt had but i'm afraid this causes to much drag for this fixed wing UAV.

Looks like this drone is not designed to carry any camera outside of the body. But there is some free space at the nose, probabry you could make some mount for a forward looking camera. In any case, the best FPV mode of this themal camera that I can suggest you is 90g within dimention of ~50x50x50mm.

[rockwell]

You are right, normally this drone carries a fixed camera looking straight down.
As this drone is launched from a catapult i can't atach a gimbal to the belly, but i can modify the nose section so it can hold a small pan-tilt head.
Again, weight is not the problem, it's size that matters.
But don't focus to much on my wishes, I can barely wait to get this camera working.

[VGN]

Finally released the first version of HyperBus memory controller. The migration from Spartan-6 was not so easy, as I expected. I had to rework quite a lot of things, and spent much more time, than I planned, but this controller is really one of the most important and critical parts of this project.

I created a special thread about this memory controller. If you have any questions or suggestions, consider using this thread:  https://www.eevblog.com/forum/fpga/openhbmc-open-source-axi4-based-hyperbus-memory-controller/

Link to the repo: https://github.com/OVGN/OpenHBMC

[zxase258]

amazing project!
It clearly shows your dedication in every aspect of the complete project.
The lovely designed Focus mechanism.
The Housing.
The Software and even the PCBs are designed with love!

I really appreciate this.

Hi! Many thanks!

You are reading the sensor out with 60fps?

Yes, the raw 14-bit per pixel data is comming out at 60fps. I left in my previos posts a link to my google drive with a video, captured right from ther sensor.
Here is the link: https://drive.google.com/file/d/1QfK6TBKxTnJjb9-Vjo1R4MsX56F_mhFq/view?usp=sharing
I advise you to download (~180MB) this video before viewing, as not smart youtube downgrades video quality too much.

How low can the clock for reading go, so it i spossible to readout at 10fps? or ist just internally fixed to 60fps?

We don't have and I believe will never have access to datasheets of this core, so I actually don't know that. Original board is forwarding clock at 73.636MHz, I'm using the same value. You will be able to experiment with clock rate, as soon as the HDL-design will be published.

On the other hand, I don't think that you really need it. If you would like to integrate this camera with any other hardware, you can choose USB, HDMI, AV or a low speed interface access over GPIO lines at X9 connector of the P-board. I have plans to implement a SPI interface over GPIO line, so that you can control the camera and readout the video data at any FPS you would like.

For what is the CMD Pin on the sensor? 1 Wire? Any information on that?

Forget about any standard interface with this FPA, everything is proprietary, the CMD interface as well. There are a lot of thing to tell about, I'm preparing a special manual for this. Well, the short answer, this is a 1-bit command line, synchronous to the main clock (73.636MHz). It is used to control the sensor. According to my investigations, a special 22 byte command is sent to the sensor to get each new frame. First two bytes of the command are 0x3F and 0xC0. I'm 99.9% sure, that this is a preamble that helps the core to detect a new command. Why? There is no any other strobe to validate the incomming command, 0xC0 = ~0x3F, very likely to be a preamble. Also, the core does not stream anything if I flip any bit in these two bytes. Other 20 bytes keep some special data to control the sensor. Unfortunatelly, the command fields are not byte aligned, that makes investigations a bit harder. There are a lot of bits that do not affect anything. Also a command for one sensor may not fit another, as themal array parameters are very different from core to core. Though I don't know what all the command fields are doing, I could find some common command pattern that should fit all sensors and a single special field that you can quite easily tune to get the image. I have tested this algorithm on my three FPAs and could make all of them working well. I'm sure that the more people will play with this sensor, the faster we will find out the function of each command field. Another way is to reverse engineer the firmware and get this data from original EEPROM. But, I'm not sure that this is a good idea, as this is an intellectual property of Avtoliv/FLIR.

Hello VGN, in the FPA timing I measured, there is another signal I don't understand its function. This signal is temporarily called "PIN6", and it appears together with the FPA data output PIN2 and PIN3. I don’t know what it does. Is it “exposure/integration” control?
In addition, from the rising edge of FSYNC (DATA IN) to the rising edge of the first data output, there are a total of 200 clocks. I think FPA samples on the falling edge of the clock.
I got a FLIR TAU324, it uses ISC0601B detector, I will measure the timing of ISC0601B. I think I also need an "FPA protocol analyzer". Specifically, I need an FPGA to monitor the commands sent by the camera to the FPA, and then send these commands to the computer to analyze how the camera configures the FPA. However, I am still new to FPGA, maybe I should ask my friend to help me do this.

73 BI4LBK.

[VGN]

Hello VGN, in the FPA timing I measured, there is another signal I don't understand its function. This signal is temporarily called "PIN6", and it appears together with the FPA data output PIN2 and PIN3. I don’t know what it does. Is it “exposure/integration” control?

If you are asking about ISC0901B0, your "PIN6" looks like to be a pixel BIAS value, check out schematic of the M-board in my repo. Controller sends 7-bit (in fact only 6-bit are significiant) LSB-first BIAS values for each pixel, each row. Approximate value is about ~0x25.

In addition, from the rising edge of FSYNC (DATA IN) to the rising edge of the first data output, there are a total of 200 clocks. I think FPA samples on the falling edge of the clock.

Is FSYNC (DATA IN)  a CMD lines in my terms? Anyway don't focus at the number of the clocks of the data output relative to the command. There is a parameter in the command that determines this value.

Concerning to the sampling edge, you are right, sensor works at the falling edge. Check out attachments. cmd_to_clk - is command or bias comming out of the controller. data_to_clk is one of the data lines comming out of the sensor.

That would be very cool if you get ISC0601B oscilloscore timing. I think this ISC0601B and ISC0901B0 are very common.

[tisher]

Car/vehicle parts

NV0  (flir pathfindir)

(12 pin connector with composite Video-out)
334-0008-00P
334-0005-00S
334-0001-00

NV1 (BMW):

(12 pin connector, no composite Video-out)
66546988779
66549134267
66549141606
66549132753
66546988774
66549125733

NV2:

(6 pin connector)
4H0980552
9205899 (66549205899)
9175668 (66549205899)

NV3:

(4 pin connector)
A2229057307
A2229052805
A2228201597
4G0980552A
9322653 (66549322653)

NV4:


btw, wat's lens dimensions in NV3?

[LesioQ]

NV3 lens' clear aperture is 15mm. Objective is of a two-lens design.
Lots of cost-saving on NV3, including Ge protective window thickness being 1/2 NV2.

[tisher]

so 'only' need to find focal length for first lens 

[mikeselectricstuff]

I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.

A few years ago, after I did a teardown video on one of these, someone emailed me, with some info they'd reverse-engineered, but didn't want to publish due to ITAR concerns.
The main point was that they had found that the original FPGA used a Microblaze core, and it was easy to find the code image in the flash memory, and after disassembling with IDA, they managed to reverse-engineer the handshake protocol used to authenticate the camera with the control box.
There might be some useful insights in to how any sensor-unique data is stored.
 

[Ismsanmar]

I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.

A few years ago, after I did a teardown video on one of these, someone emailed me, with some info they'd reverse-engineered, but didn't want to publish due to ITAR concerns.
The main point was that they had found that the original FPGA used a Microblaze core, and it was easy to find the code image in the flash memory, and after disassembling with IDA, they managed to reverse-engineer the handshake protocol used to authenticate the camera with the control box.
There might be some useful insights in to how any sensor-unique data is stored.

Are you talking about this?:
https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/

[LesioQ]

so 'only' need to find focal length for first lens 

What's the purpose of it's focal length ? For f-stop calculation You need effective f.l. of complete objective.

[mikeselectricstuff]

Had a look at the schematic. Debouncing of the buttons will be done in VHDL via a filter in the FPGA? In the microcontroller unit it is an annoying task to debounce something.

Buttons pull-ups and debouncing capacitors are at the P-board (peripheral). Of course, there is additional debouncing logic in the HDL design. Agree, filtering buttons by internal mcu is a poor idea.

Debouncing entirely in software is trivially easy - you simply sample the buttons every 20-40mS or so.

[tmbinc]

I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.

A few years ago, after I did a teardown video on one of these, someone emailed me, with some info they'd reverse-engineered, but didn't want to publish due to ITAR concerns.
The main point was that they had found that the original FPGA used a Microblaze core, and it was easy to find the code image in the flash memory, and after disassembling with IDA, they managed to reverse-engineer the handshake protocol used to authenticate the camera with the control box.
There might be some useful insights in to how any sensor-unique data is stored.

Hi Mike, yes, that was me. I've since then published my findings - in slightly redacted form - in https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/.

However, my question specifically is a.) about the NV3 family, and b.) more specifcally the FLIR PathFindIR II, and how it differs from the automotive NV3 that we've all seen.

[VGN]

Hello everyone, some updates...

Sorry again for this delay... but this 2020 year is a nightmare in absolutely every aspect of life.
But I try to do my best to keep developing the project no matter what happens.

Back on project:

1. As the hardware development is almost done, I have ordered electronic parts for the new hardware revision last monday, hope to get them in 2-3 weeks (yes...too long, but there is no any other way to speed up this process).

2. I have finally finished all shematics and routed all PCBs, including recomendations that I received from you. The whole set of schematics is published, chekout my repo.
PCB board production time is about 1 week, but I'm going to ordered new boards next week to get both boards and components simultaneously. If you have any suggestions according to newest schematics, please let me know, this week is the last chance to make any changes.
One more link to the repo: https://github.com/OVGN/OpenIRV/tree/master/docs/schematics

3. All firmware and HDL sources will be published along with porting to new hardware. As soon as I get sure that new hardware is working well, I will start beta production for testers and developers.

[dmendesf]

Congratulations, your Altium skills are top notch.

[VGN]

Hi! Some updates.

Good news, I received most part of active components, i.e. power supply ICs and the FPGA! Now waiting for some passive smd stuff from another supplier. New PCBs were ordered several days ago. Hope to start assembling everything in a week or a little later.

Meanwhile, I decided to make some improvements to the focusing system. First, I have modified the motor mount. Earlier I noticed that motor rigid connection to the board is not a good idea, as motor vibrations sound is amplified by the focus PCB. I decided to use a special silicone cover between the motor and the motor mount. I haven't printed this parts yet, so let me know if you have ideas of impromevents.

The second modification is the focus motor itself. Somebody noticed that focus adjustment is really slow. Yes, this is true. This is because I have selected a motor with very high gear ration and low rpm (1:136 @240RPM) as I was not sure if this tiny motor can apply enough force to move the lens. There is another type of this motor with lower gear ration and higher RPM (1:26 @1200RPM). I haven't ordered it yet, but...I decided to modify one of my motors. If you take it apart, you will find 3 stages of tiny planetary gear. I removed one the stages, shortened the motor housing and put all back together. Now it is running much faster! Checkout the difference at video:
https://youtu.be/GuIjXj8-yEI

Congratulations, your Altium skills are top notch.

Thanks, though there so many thing to learn)

[rockwell]

New focus speed is for my purpose now fast enough.

[VGN]

New focus speed is for my purpose now fast enough.

One more video of focusing right from the thermal camera. This is not autofocus yet, I'm adjusting the focus manually and not accurate enough sometimes. The autofocus will do this job much better that me.

Youtube collapces video quality too much, that's why download this video before viewing, it is only ~39MB.
https://drive.google.com/file/d/1_MNzznl6It1Ix80HIM0NVI9G4iD7iLla/view?usp=sharing

[VGN]

While waiting for PCBs and components, I decided to reveal some future plans.

ISC0901 is a pretty good 336x256 60FPS sensor. This characteristics is enougth for quite a lot of different applications.
But I have one a bit crazy idea. I would like to significially increase the resolution. As you know, the whole hardware was
designed in a way to support any other sensor (I mean, configurable power lines, a lot of high performance IOs, RAM, etc.)
But unfortunatelly this is really hard to find cheap high resolution sensor...

If you look at the SEM photos of LWIR array, you could see a quite interesting pixel structure. Note, this is a photo of the ISC0601, not ISC0901.
The main difference is the pixel pitch. ISC0601 has 25um pixel pitch, while ISC0901's pixel pitch looks like to be 17um.
I could not find SEM photos of the ISC0901 sensor pixel array, except one photo of the beam holding "via", that proves somehow that ISC0901 has the same
pixel pattern as ISC0601.

If you have or saw somewhere the SEM photos of ISC0901 pixel array, please share it here, that will be very helpful!

The pink area is actually the active pixel, this is a resistor that is supported by two special thin beams, that allow to make
thermal insulation of the pixel from the die and other pixels. The radiation comming from the lens heats up each pixel.
The resistance of the each pixel changes according to the radiation level. This resistance value is calculated by internal ADC,
so that's how we get the pixel value.

Due to technology limitations, the active pixel area is only about 21% of the whole FPA area. There are huge gapes between pixels, that are not doing any useful work.
So simply shifting the array for a 1/2 of pixel pitch value in X and Y direction, we could increase the resolution from 336x256 to 672x512.

But this is also not a limit yet, as making more accurate steps (1/4 or 1/8 pixel pitch) along with special deconvolution algorithms it is possible
to increase the resolution even more! The last attached photo shows the result of x5 super resolution image reconstruction, based on 20 low resolution frames. Looks quite impressive for me.

I'm not sure, but I hope to get a megapixel camera this way))

[zrq]

If someone in the EU have a faulty sensor, please send it to me. I'll be happy to get some SEM images. (I'll try my best  to persuade my advisor to allow me to do that)

[VGN]

If someone in the EU have a faulty sensor, please send it to me. I'll be happy to get some SEM images. (I'll try my best  to persuade my advisor to allow me to do that)

That will be really nice to get SEM photos, but not mandatory, as we know the most important value - the pixel pitch of 17um. FLIR’s Tau2 camera core has 17 μm pixel pitch and we know that it is based on ISC0901.

Probably knowing dimentions of the pixel's active area and gaps around it will be helpful for proper enlarged image calculation. I think this the only reason to sacrify single lwir core and get SEM photos.

[VGN]

Now two words about how I'm going to implement this pixel shifting.

Well, we are talking about micrometer precision level of displacement. I know that there is a trick of using handheld motion effect,
that allows to get this small sensor displacements (Fluke implemented this feature on some of their thermal imagers).
But don't like it at all, as handheld motion based super resolution (SR) is not accurate enough and also will not work in case of tripod mounting.

I'm going to use piezo actuators along with a custom XY-stage.

You can find a lot of info about piezo actuators and the physics of  inverse piezoelectric effect. In two words this is crystal that can enlarge
for some small value if you apply some voltage. Piezo actuator behaves like a common capcacitor, and displacement value is equivalent to the charge level.
The dependency between the voltage and actual displacenent value is linear. It is possible to get even nm accuracy if desired.

ISC0901 has 17um pixel pitch, so we should select an actuator with a displacement value > pixel_pitch/2. I have selected AE0203D18H18DF, that has
max displacement level of 19.0 ±2.0um at 150V. Actuator dimentions are 18x3x2mm. The generated displacement force just blows my mind, it is about 200N (yes, two hundred).

I made some sketches of the XY stage and also some simulations. The thickness of the moving platform beams is about 0.8mm. If made of 6061 aluminium we need to
apply only 10N to displace the platform for about 16um, while the actuator can generate 200N.

Note that the movement level at attached gifs is amplified a lot, this is not the real value, as 18um displacement value is so small, that this is simply impossible to see it with an unaided eye.

I have never developed anything using such kind of XY-stages and piezo actuators, so suggestions and critics are welcome.


Advantages:

1. It is possible to significially increase the resolution. The gaps between active pixel area are so big, that pitch/2 shift will give a clear x4 resolution. More accurate shifting steps along with deconvolution image processing will increase the resolution even more, I hope to get a megapixel.
2. The construction will be fully backward compatible with whole other design, including focusing board.
3. This XY stage can be used with any other sensors, if attached over special adapter. New LWIR cores have even lower pixel pitch of 12um and we can still increase the resolution.

Disadvantages:

1. The piezo actuator, this piece of... shi ceramics... costs around 100$. Yes, it is even more expensive that the FPGA. And...we need two of them for X and Y axis. 
2. We will have to deal with high DC voltages under 150V, well this is a bit more dangerous than usual. There are special ICs for driving actuators with integrated dc-dc boosters that allow to get this high voltage.
3. Not sure, but probably the whole construction will be a bit more fragile, expecially because of the ceramic actuators, though I'm sure you wouldn't let it fall down)

[Vipitis]

Pixel shifting can be found in Sony cameras and Hasselblad. There are many thread in this subforum that touch on the idea and some did tests. Pixel shifting is not viable for video, but a slow photomode for high resolution stills would be a great feature I would love to have.
The claims about active area vs Pixel pitch Sounds a little odd. We know that Leonardo/DRS has a special pixel design that is a lot more area efficient than the one shown here, using "micro umbrellas". It might be the sole reason why they managed to 10μm somewhat useable.

Also a thought: while you shift the sensor, why not include a whole 5 axis platform for in body stabilizion? Sony uses that for their pixel shifting as well. And they do capture like 12 individual frames on the a7RIV to get 120MP and no interpolation.

Do you hope to shift by perfectly 1/2 the pixel pitch or would 1.5 also work?  If you go beyond 4x this would require getting 0.33,0.66 or even 0.25, -0.25 too. While it might seem that perfect precision makes this sound possible - the real world has limitations in holding perfectly still. So a potential photomode could also include stacking of same pixels shifted by 1 to reduce noise.


There is great development done in this technology for surveillance satellites, so it's very much inaccessible.

[Fraser]

In case this paper is of use/interest.....

Vibration induced Resonance and the effects of G forces in microbolometer pixels ......

https://www.jvejournals.com/article/15829/pdf

Fraser

[mikeselectricstuff]

So simply shifting the array for a 1/2 of pixel pitch value in X and Y direction, we could increase the resolution from 336x256 to 672x512.

Bear in mind you may need a much better ( more expensive) lens to get improved resolution. If it is the case that it's only sampling a fairly small percentage of the sensor area, then you would see noticeable aliasing artifacts (e.g. jaggies on straight edges) unless the existing optics system has been designed to slightly blur the image to antialias it. It may also be the case that there is some post-processing going on as well to reduce aliasing effects.
I'm sure there is some scope to improve resolution by moving the sensor and applying some clever maths, but I'm not convinced the results would be spectacular.
You can see this effect in this video where I was under-sampling a video camera :
https://youtu.be/rQYByorpoFk?t=869

[SilverSolder]

Wouldn't it have to be a pretty terrible lens to not be able to resolve  336x256, for example?

[mikeselectricstuff]

Wouldn't it have to be a pretty terrible lens to not be able to resolve  336x256, for example?

Yes, but it is probably engineered to that resolution, giving just the right amount of blur to avoid aliasing artifacts at that resolution

[VGN]

Pixel shifting can be found in Sony cameras and Hasselblad. There are many thread in this subforum that touch on the idea and some did tests. Pixel shifting is not viable for video, but a slow photomode for high resolution stills would be a great feature I would love to have.

Yes, but as I know pixel shifting in common cameras is mostly used to breaktrough the limitation on Bayer filter, i.e. shifting the sensor you can get true R,G,B values for each pixel, so there is no need to make debayering, as we already have all color components. I don't really know, why it is called super resolution, as it is not that, we don't get more pixels:

Pixel shifting is not viable for video, but a slow photomode for high resolution stills would be a great feature I would love to have.

True, but piezo actuators are extremely fast. So theoretically it is possible to shoot a video, though the FPS will be n^2 times lower, where n is a factor of resolution enlargement.

The claims about active area vs Pixel pitch Sounds a little odd. We know that Leonardo/DRS has a special pixel design that is a lot more area efficient than the one shown here, using "micro umbrellas". It might be the sole reason why they managed to 10μm somewhat useable.

That was news to me! So it turns out that each "micro umbrella" just heats up the same VOx film, which is placed under it, supported by the same know thermal isolation pattern? And we have almost no pixel gaps. This is really incredible! Checkout attachments.

Also a thought: while you shift the sensor, why not include a whole 5 axis platform for in body stabilizion? Sony uses that for their pixel shifting as well. And they do capture like 12 individual frames on the a7RIV to get 120MP and no interpolation.

That is going to be too complex and expensive...Moreover, no idea what to do with 5 axis.

Do you hope to shift by perfectly 1/2 the pixel pitch or would 1.5 also work?  If you go beyond 4x this would require getting 0.33,0.66 or even 0.25, -0.25 too. While it might seem that perfect precision makes this sound possible - the real world has limitations in holding perfectly still. So a potential photomode could also include stacking of same pixels shifted by 1 to reduce noise.

I hope to shift with much smaller steps that 1/2. Anyway this is a matter of real experiments, as I expect some limitations and problems. I also should find some way for step calibration.

[VGN]

Vibration induced Resonance and the effects of G forces in microbolometer pixels ......

Thanks, Fraser. That should be taken in account, but I think that vibration frequency caused by pixel shifting actuators will be far away from the sensor's pixel resonance frequency.

Bear in mind you may need a much better ( more expensive) lens to get improved resolution.

Thanks, Mike, I didn't thought about that. Well, that may be a huge problem. I'm not really good with optics, but original FLIR E8 lens is much worse that this one, but it was also designed for the right same ISC0901 sensor.

Do you have any ideas how to check this lens capabilities?

[_Wim_]

1. The piezo actuator, this piece of... shi ceramics... costs around 100$. Yes, it is even more expensive that the FPGA. And...we need two of them for X and Y axis. 

I would give these a try: https://nl.aliexpress.com/item/32874018830.html?spm=a2g0o.productlist.0.0.2b033905e3vloB&algo_pvid=23e2f0b8-e665-4619-bd80-5c8e64b3aa5c&algo_expid=23e2f0b8-e665-4619-bd80-5c8e64b3aa5c-11&btsid=0b0a555c16054735796882807e75f6&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_

180µm for 100V, should be around 18µm for 10V, and as you are 3D printing your housing and absolute accuracy is not required for this application, I think these would be sufficient.

[VGN]

I would give these a try: ...
180µm for 100V, should be around 18µm for 10V, and as you are 3D printing your housing and absolute accuracy is not required for this application, I think these would be sufficient.

Hm... the characteristics look quite fantastic for this dimentions. Have you ever used this parts?

For example, both different well known manufacturers, and the parts with common dimentions have commom characteristics:
Kemet's AE0203D18H18DF from here: https://ru.mouser.com/datasheet/2/212/1/KEM_P0101_AE-1518874.pdf
PI's P-882.51 from here: https://static.pi-usa.us/fileadmin/user_upload/physik_instrumente/files/datasheets/P-882-Datasheet.pdf

I have absolutely no experience with piezo actuators, but chinese ones look really strange. Maybe I'm wrong, not sure this is truth. We need anyone how can prove that.

[Bill W]

Wouldn't it have to be a pretty terrible lens to not be able to resolve  336x256, for example?

Not at all.  The real issue is the lens 'circle of confusion' versus the pixel pitch.  Lots of appropriate maths at
https://www.dofmaster.com/dofjs.html

Our thermal lenses score badly due to the large apertures used and the expense of processing or multiple elements.
A simple 2 element 'spherical' design (50mm f/0.7) meant for a Pevicon tube really does not resolve well on 17µm pitch.

Bill

[bap2703]

Do you have any ideas how to check this lens capabilities?

At first you can probably do a simple check:
Can you get a sharp image of a point or edge source?

The real thing is called measuring the modulation transfer function (MTF) of the lens.
Basically you describe an image in terms of spatial frequencies: how fast the luminosity is changing quickly across pixels.
Details equal to high frequencies for example. Since you played with frequency domain filtering of images you'll quickly grasp how that works.
Think of lenses like low-pass filters: in order to get more details, you need to grab more high-frequencies.
Lens designers would probably match the cut-off to the targeted sensor pitch if it allows to build it cheaper.

You can probably get to half the pitch without too much trouble (or software sharpening), but more than that you'll probably notice the lens limitations.

You can read more there:
https://www.umicore.com/en/newsroom/the-evolution-of-lens-designs-for-12micron-uncooled-lwir-detectors/

[_Wim_]

I would give these a try: ...
180µm for 100V, should be around 18µm for 10V, and as you are 3D printing your housing and absolute accuracy is not required for this application, I think these would be sufficient.

Hm... the characteristics look quite fantastic for this dimentions. Have you ever used this parts?

For example, both different well known manufacturers, and the parts with common dimentions have commom characteristics:
Kemet's AE0203D18H18DF from here: https://ru.mouser.com/datasheet/2/212/1/KEM_P0101_AE-1518874.pdf
PI's P-882.51 from here: https://static.pi-usa.us/fileadmin/user_upload/physik_instrumente/files/datasheets/P-882-Datasheet.pdf

I have absolutely no experience with piezo actuators, but chinese ones look really strange. Maybe I'm wrong, not sure this is truth. We need anyone how can prove that.

I have no experience with those yet. I did an experiment with a low cost piezo buzzer (with the intention of building an Scanning Fabry-Perot Interferometer like http://repairfaq.cis.upenn.edu/Misc/sale/sfpiins1.htm).

This is the type of piezo I am talking about: https://nl.aliexpress.com/item/4000120679339.html?spm=a2g0o.productlist.0.0.5cde25c3H0l9jK&algo_pvid=39a8075f-6e25-492a-911d-34217e99d7f2&algo_expid=39a8075f-6e25-492a-911d-34217e99d7f2-3&btsid=0bb0623416055593233587628ed6a4&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_

These have a ceramic layer of only approximately 0.2mm thick. It was connected directly to my signal generator with a 10Vpp output. A mirror was glued to the piezo, and distance was measured with a DIY interferometer (details here: http://www.repairfaq.org/sam/uMD1/). This gave me a little less than 250nm pp. Taking into account the ceramic material was only 0.2mm thick, this would mean 6.25µm pp for a 5mm thick material. This is below the 18µm spec for 10V, but if similar specs, with a higher voltage, 17µm must be easily achievable.

To be sure, I also ordered a set (https://nl.aliexpress.com/item/32952294614.html?spm=a2g0o.productlist.0.0.4da739052Pa8qc&algo_pvid=aa231da8-f9e6-4c2d-97a4-be556ee5db3d&algo_expid=aa231da8-f9e6-4c2d-97a4-be556ee5db3d-0&btsid=0bb0624516055606928738744e86ab&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_) and will test them the same way.
 

[_Wim_]

The results above need to be divided by 2, as I used a plane mirror interferometer setup instead of my "usual" linear setup with retro reflector, but did not change the settings accordingly in µmd1.

So this means with a 10V signal that kind of piezo only does 125nm instead of the 250nm shown in the result plot.

[LesioQ]

The ultimate limit will be Airy disk dia, i.e. physics.

http://www.calctool.org/CALC/phys/optics/spot_size

[VGN]

At first you can probably do a simple check:
Can you get a sharp image of a point or edge source?

Sure.

Experiment #1:
The distance to the object is about 5m. I used an aluminium tape to make a sharp edge. I was playing with focus and camera position for ~20 minutes. This is the best edge sharpness, that I could achieve. I started to feel lens limitations)

[VGN]

At first you can probably do a simple check:
Can you get a sharp image of a point or edge source?

Experiment #2:
The distance to the object is about 0,4m:

[VGN]

The real thing is called measuring the modulation transfer function (MTF) of the lens.
Basically you describe an image in terms of spatial frequencies: how fast the luminosity is changing quickly across pixels.
Details equal to high frequencies for example. Since you played with frequency domain filtering of images you'll quickly grasp how that works.
Think of lenses like low-pass filters: in order to get more details, you need to grab more high-frequencies.
Lens designers would probably match the cut-off to the targeted sensor pitch if it allows to build it cheaper.

bap2703, thank you very much, that was really helpful! Looks like I started somehow understand what is going on... 
So, it turn out that using pixel shifting I just actually decrease the sensor pixel pitch. And if I want to keep the image sharpness, I should increase the lens "bandwidth" (as it is an optical low-pass filter)

From the article:
Since the lens performance is essentially diffraction limited, there is only one way to increase the MTF of the lens so that it remains constant for the higher spatial frequency. That is to make the lens faster in the ratio of the change in pixel pitch.

Ok, so if I got it right, we need a faster lens if we want to decrease the pixel pitch. Probably, first we should determine the lens f-number.

You can probably get to half the pitch without too much trouble (or software sharpening), but more than that you'll probably notice the lens limitations.

If I'm not wrong, not only lens limitations. As the pixel itself is not a spot, i.e. it has some dimentions, we will also face with pixel overlap at very small steps.

[VGN]

I have no experience with those yet. I did an experiment with a low cost piezo buzzer (with the intention of building an Scanning Fabry-Perot Interferometer like http://repairfaq.cis.upenn.edu/Misc/sale/sfpiins1.htm).

This is the type of piezo I am talking about: https://nl.aliexpress.com/item/4000120679339.html?spm=a2g0o.productlist.0.0.5cde25c3H0l9jK&algo_pvid=39a8075f-6e25-492a-911d-34217e99d7f2&algo_expid=39a8075f-6e25-492a-911d-34217e99d7f2-3&btsid=0bb0623416055593233587628ed6a4&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_

These have a ceramic layer of only approximately 0.2mm thick. It was connected directly to my signal generator with a 10Vpp output. A mirror was glued to the piezo, and distance was measured with a DIY interferometer (details here: http://www.repairfaq.org/sam/uMD1/). This gave me a little less than 250nm pp. Taking into account the ceramic material was only 0.2mm thick, this would mean 6.25µm pp for a 5mm thick material. This is below the 18µm spec for 10V, but if similar specs, with a higher voltage, 17µm must be easily achievable.

To be sure, I also ordered a set (https://nl.aliexpress.com/item/32952294614.html?spm=a2g0o.productlist.0.0.4da739052Pa8qc&algo_pvid=aa231da8-f9e6-4c2d-97a4-be556ee5db3d&algo_expid=aa231da8-f9e6-4c2d-97a4-be556ee5db3d-0&btsid=0bb0624516055606928738744e86ab&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_) and will test them the same way.

Nice project!

This gave me a little less than 250nm pp. Taking into account the ceramic material was only 0.2mm thick, this would mean 6.25µm pp for a 5mm thick material. This is below the 18µm spec for 10V, but if similar specs, with a higher voltage, 17µm must be easily achievable.

Agree, that makes sense.


Found this "datasheet" for AL1.65X1.65X5D-4F, according to it only 3,8um@90V
https://img.alicdn.com/imgextra/i2/71977092/TB2wz4ybNaK.eBjSZFAXXczFXXa_!!71977092.png

To be sure, I also ordered a set and will test them the same way.

Anyway, I will be very grateful if you share your results with us!

[Fraser]

Just for information, the Autoliv camera is basically a variation on the TAU and TAU 2. Specifications for the various lens options are available. In my experience modern lenses for the TAU cameras are all of adequate resolution to work well with a VGA 17um microbolometer. If you look at suppliers, such as Ophir, it is clear that modern thermal imaging lenses are more than capable of VGA resolutions on 17um pixels and even XGA on 12um pixels.

I would be very surprised if the lens installed in a TAU, or TAU based core was not capable of at least VGA on 17um pixels. Lens manufacturing has become finely honed to offer affordable thermal imaging lenses capable of good resolution. Even Chalcogenide IR Glass lenses can achieve such good performance these days provided the lens designer does his job correctly. A manufacturer will normally design a lens set that will work with a large range of microbolometers and the Ophir catalogue shows this. XGA 12um capability is now common. I attach a link to the TAU brochure and the Ophir catalogue for information. How I would love to have free access to the Ophir range of lenses  . Take a look at the 65221 19mm lens specification on page 52. That is not a super expensive lens yet it is adequate resolution for 1024 x 768 12um pixels. That 19mm lens is used in the ThermApp camera.

http://52ebad10ee97eea25d5e-d7d40819259e7d3022d9ad53e3694148.r84.cf3.rackcdn.com/UK_Z24-FLIR_Tau2_Family_Brochure_DS_CAT-08.pdf

http://52ebad10ee97eea25d5e-d7d40819259e7d3022d9ad53e3694148.r84.cf3.rackcdn.com/UK_Z24-FLIR_Tau2_Family_Brochure_DS_CAT-08.pdf

From my perspective, I just cannot see a modern thermal imaging lens manufacturer going to the effort of making a poorly performing lens that cannot cope with VGA 17um pixels as a minimum. Where the mechanical limits permit, I would expect most modern lenses to cope well with XGA. It is true, as Bill_W has stated, that older thermal imaging cameras that used much larger pixels and arrays that were QVGA or smaller did have resolution limits associated with the lens block. That was an era when pure Germanium lenses were cut on mono point diamond lens forming lathes. Such was a very expensive process and remains so to this day. The lenses tended to be cut to the requirement rather than to needlessly exceed it at additional cost. Times have changed with the introduction of Chalcogenide IR glass lenses that are precision moulded and more advanced pure Germanium cutting processes. Thermal camera lens manufacturing has had to advance to keep in step with newer, smaller and higher resolution imaging sensors.

I caveat all of the above by saying that a manufacturer who wants a camera built down to a certain production cost may still cut corners on the lens part of the design. It is still possible to have relatively poorly performing thermal imaging lenses produced where image quality is not a high priority. Such lenses are commonly made for less demanding applications such as industrial IR temperature sensors so lens manufacturers do have relatively low grade lens production facilities.

FLIR went as far as using a ‘printed’ diffractive silicon lens in their Lepton core and that lens is resolution limited. How I wish there was a conventional lens option for that core.

Fraser

[bap2703]

Experiment #2:
The distance to the object is about 0,4m:

I'd say you lens looks definitely adequate to take advantage of superresolution.

If I'm not wrong, not only lens limitations. As the pixel itself is not a spot, i.e. it has some dimentions, we will also face with pixel overlap at very small steps.

You are right and that's mathematically all the same:
- the response of the lens is an image where every source point is spread (and that's called the PSF, point spread function)
- a pixel has a spatial response too, be it from it's physical size or some "antennae" effect (you posted picture of pixels with a grid structure for example)

When you know them, you can to some extent compensate their effects.
That's called deconvolution and it's used everywhere.

[_Wim_]

Found this "datasheet" for AL1.65X1.65X5D-4F, according to it only 3,8um@90V
https://img.alicdn.com/imgextra/i2/71977092/TB2wz4ybNaK.eBjSZFAXXczFXXa_!!71977092.png

  ? That could be correct also of coarse! Anyway, it will be fun to play with, and I need to "use" my interferometer...

Found this "datasheet" for AL1.65X1.65X5D-4F, according to it only 3,8um@90V
https://img.alicdn.com/imgextra/i2/71977092/TB2wz4ybNaK.eBjSZFAXXczFXXa_!!71977092.png
Anyway, I will be very grateful if you share your results with us!

 
Will do, but as we all now, it can take a while until the Chinese Santa Clause comes to our door...

[VGN]

I'd say you lens looks definitely adequate to take advantage of superresolution.

I hope that very much too. But I also want to try to prove this theoretically if it is possible.

The ultimate limit will be Airy disk dia, i.e. physics.

http://www.calctool.org/CALC/phys/optics/spot_size

Ok, lets calculate this Airy disk dia.

First, let's get the f-number of the lens.
According to this datasheet (table 6.1): http://www.safetyvision.com/sites/safetyvision.com/files/FLIR_PathFindIRII_User_Guide_1.pdf
The lens focal length is f = 19mm (seems to be true). The aperture circle dia D = 15mm. In this way f-number = f/D = 19/15 = 1.26(6). The closest standard value is 1.25.
So, let's say that we have a f/1.25 lens.

Finally for LWIR spectral band of (8 - 14um) we have an Airy disk dia of (21.35 - 42.7um). Well...even smallest value of this range is much larger than the actual pixel size for a 17um pitch technology...

Again, I'm not good enough with optics, but the result looks a bit weird for me, I didn't expect such values. Any ideas what am I doing or interpreting wrong? 

[Fraser]

Your calculations look correct to me.

I did not include the link to the Ophir catalogue in my earlier post, sorry.

Look at the 19mm lens on page 52 and enter its specs into an on line Airy Disk calculator (as I did). It verifies your calculated result.

https://applied-infrared.com.au/images/pdf/Ophir_Thermal_Imaging_Lenses_Catalog.pdf

I believe you are showing the limitations of the optics used in thermal imaging when small pixel sizes are involved. This may go some way to explain why microbolometer arrays that use 12um pixels struggle with contrast.

I believe anti-aliasing and dynamic contrast enhancement techniques are employed to improve the contrast in images produced by thermal cameras. It would appear that any core that uses a pixel size smaller than 25um is battling the Airy Disk limits of the commonly available optical systems.

Thank you for highlighting this situation as I have not previously looked at it for thermal lenses 

Airy Disk Calculator :

http://www.calctool.org/CALC/phys/optics/f_NA

For the referenced 19mm Ophir lens the Airy disk is 27um at 10um wavelength.

Fraser

[bap2703]

The game doesn't end with calculating the airy disc diameter ! That's a common mistake.

What do you want to do next?
--> sample it at more than just one pixel per airy diameter!
Because you're not really interested in sampling the airy disc with one pixel, but sampling the tiny contrast (in amplitude AND space) resulting from merging two close diffracted spots.

You can sample at a pitch much less than the airy diameter.

I found that document that appears to explains it quite well (I had a quick glimpse at it but it seems good).
https://www.researchgate.net/publication/336775082_Resolving_some_spatial_resolution_issues_-_Part_2_When_diffraction_takes_over

That one says a pitch of 1/6 of the airy diameter !

[VGN]

Thank you for highlighting this situation as I have not previously looked at it for thermal lenses 

Thanks to LesioQ and our cruel, but beautiful universe! 

The game doesn't end with calculating the airy disc diameter ! That's a common mistake.

What do you want to do next?
--> sample it at more than just one pixel per airy diameter!
Because you're not really interested in sampling the airy disc with one pixel, but sampling the tiny contrast (in amplitude AND space) resulting from merging two close diffracted spots.

You can sample at a pitch much less than the airy diameter.

I found that document that appears to explains it quite well (I had a quick glimpse at it but it seems good).

First, thank you for this article, it is really mind changing. I recommend everyone to read it.
Second, you are absolutely right, we can sample at a small pitch to get maximum possible spatial resolution that current lens is capable to provide. I found quite nice picture from one astronomy forum, that shows the advantage of high accurate airy disk sampling.

That one says a pitch of 1/6 of the airy diameter !

Here is a quote from the article, explaining why we are talking about 1/6 of the airy diameter for those, who didn't find it:

Since we know from the Rayleigh criterion that Δxmin equals the radius of the Airy disc, it follows that the best
‘match’ between sensor characteristics and the optics is achieved when the detector pitch p equals one‐third of
the Airy disc radius. In other words: three pixels suffice to resolve the Airy disc radius entirely. The reasoning
then goes that above that threshold, the optics diffraction limitation kicks in. So digitising the radius of the PSF’s
central part with more than three pixels does not improve the final spatial resolution of the image because the
detector only resolves diffraction blur at that point.

BTW, I didn't now, but FLIR has a special UltraMax technology, that allows to increase the resolution. It is very silimilar to Fluke's superresolution.
The most interesting moment is at 0:51, the main idea is the same, just sample the data in gaps between pixels, as active pixel area is really small.
Full video: https://youtu.be/8y2lV3nIhvc

[VGN]

Some updates.

Today received new two layer PCBs, i.e. buttons, display and the magnets holding ring! Going to assembly this boards today.

Unfortunately looks like due to covid-disaster manufacturer delays boards release, that's why main and peripheral boards will apear in my hands only at next week.

[Max Planck]

I'd say you lens looks definitely adequate to take advantage of superresolution.

I hope that very much too. But I also want to try to prove this theoretically if it is possible.

The ultimate limit will be Airy disk dia, i.e. physics.

http://www.calctool.org/CALC/phys/optics/spot_size

Ok, lets calculate this Airy disk dia.

First, let's get the f-number of the lens.
According to this datasheet (table 6.1): http://www.safetyvision.com/sites/safetyvision.com/files/FLIR_PathFindIRII_User_Guide_1.pdf
The lens focal length is f = 19mm (seems to be true). The aperture circle dia D = 15mm. In this way f-number = f/D = 19/15 = 1.26(6). The closest standard value is 1.25.
So, let's say that we have a f/1.25 lens.

Finally for LWIR spectral band of (8 - 14um) we have an Airy disk dia of (21.35 - 42.7um). Well...even smallest value of this range is much larger than the actual pixel size for a 17um pitch technology...

Again, I'm not good enough with optics, but the result looks a bit weird for me, I didn't expect such values. Any ideas what am I doing or interpreting wrong? 

One minor detail. The focal length given on a lens is a value when focusing your camera at infinity. In most cases a so called effective focal length should be used instead, also when calculating Airy disk diameter.

Max

[VGN]

One minor detail. The focal length given on a lens is a value when focusing your camera at infinity. In most cases a so called effective focal length should be used instead, also when calculating Airy disk diameter.

I'll be grateful if you explain it in more detail and attach a picture if possible. Effective Focal Length (EFL) is a bit overloaded term. The resulting focal length of multiple lenses is called EFL. Also photographers call EFL a focal length affected by the camera’s crop factor. But looks like you are talking about other things.


Assembled both boards. Everything looks to be ok. I like new buttons much more than previous ones. Magnet ring fits pefrectly, though without magnets, as I haven't received them yet.

[Vipitis]

The buttons remind me of the Blackmagic Design Micro Cinema Camera.

[Max Planck]

One minor detail. The focal length given on a lens is a value when focusing your camera at infinity. In most cases a so called effective focal length should be used instead, also when calculating Airy disk diameter.

I'll be grateful if you explain it in more detail and attach a picture if possible. Effective Focal Length (EFL) is a bit overloaded term. The resulting focal length of multiple lenses is called EFL. Also photographers call EFL a focal length affected by the camera’s crop factor. But looks like you are talking about other things.

Please look at the link below.

https://www.optowiki.info/glossary/working-f-number/

 The article is about the effective/working f-number, but basically it is about the same problem, i.e. not focusing your lens at infinity.

If you are calculating the Airy disk diameter as 2.44 x wavelength of light x f-number, you should use the working f-number, not the one given on the lens.

One more thing, stemmer imaging published several tutorials about choosing the correct lens.


Max

[Bill W]

Fortunately only a big issue in extreme close up work with long lenses.

An 8mm lens focussed down to 100mm is only reduced in illumination / field of view by 10% as the image is spread out

Bill

[VGN]

Finally received all components and PCBs!

[rockwell]

Can't wait to see the boards populated and heading to first tests.

[VGN]

Can't wait to see the boards populated and heading to first tests.

Me too)


I started from the M-board. As I have made too many changes to the power system since previous hardware revision, I decided to check power supply first, before assembling RAM and FPGA. Generally, new power system works fine!

[VGN]

M-board is fully assembled. No magic smoke, power consumption at expected level, FPGA is alive, both RAM ranks memtest is OK!

[bap2703]

That looks so professional 

[VGN]

That looks so professional 

Thanks! Doing my best)


The P-board is ready!

[dmendesf]

Seems your project is going well. When will you accept beta-testers?  Planning a crowdfunding of some sort?

[VGN]

Seems your project is going well. When will you accept beta-testers?  Planning a crowdfunding of some sort?

I'm still accepting beta-testers, checkout the very first post of this thread (have updated it for clarity), you will find some details about beta tesing. More information I will provide further.

Planning a crowdfunding of some sort?

Sure, but first I must get sure that every single part of this complex device is working properly. I really don't want anyone to face with hardware problems, even beta-testers.

[_Wim_]

Found this "datasheet" for AL1.65X1.65X5D-4F, according to it only 3,8um@90V
https://img.alicdn.com/imgextra/i2/71977092/TB2wz4ybNaK.eBjSZFAXXczFXXa_!!71977092.png

To be sure, I also ordered a set and will test them the same way.

Anyway, I will be very grateful if you share your results with us!

I received the piezo elements today. And unfortunately the "datasheet" you found was correct. Because they were so small, I could not easily use my interferometer meter, but I used a Keyence LK-G37 laser sensor instead. Signal is a bit noiser, but if they would have met their claimed spec, the Keyence should be more than capable also.

Result: with a 0 to 10 volt sine wave, I measure about a +-0.4µm signal, which matches with the 3.7µm@90V from the datasheet...

[VGN]

Result: with a 0 to 10 volt sine wave, I measure about a +-0.4µm signal, which matches with the 3.7µm@90V from the datasheet...

Thanks so much, _Wim_ , for proving the characteristics!
Yes, this is pity, but ok, at least now we know the truth.

Well, looks like best candidates are still:

Kemet's AE0203D18H18DF from here: https://ru.mouser.com/datasheet/2/212/1/KEM_P0101_AE-1518874.pdf
PI's P-882.51 from here: https://static.pi-usa.us/fileadmin/user_upload/physik_instrumente/files/datasheets/P-882-Datasheet.pdf

Though Kemet's one is avaliable at both Digikey and Mouser, I hope to find something cheaper if it is possible.

[_Wim_]

Well, looks like best candidates are still:

Kemet's AE0203D18H18DF from here: https://ru.mouser.com/datasheet/2/212/1/KEM_P0101_AE-1518874.pdf
PI's P-882.51 from here: https://static.pi-usa.us/fileadmin/user_upload/physik_instrumente/files/datasheets/P-882-Datasheet.pdf

Though Kemet's one is avaliable at both Digikey and Mouser, I hope to find something cheaper if it is possible.

Those are indeed expsive, for sure if you need at least 2 per camera.

As you doing your own mechanical design also, would integrating a lever to increase the stroke be an option?

[JanHenrik]

I really love this project, thus I got two NV3 right away! :3

Since I also like the inbuilt superresolution-stage idea, I redrew the XY-Stage as board in KiCAD and ordered a few. Mine are based on either MLCCs or those shitty AliExpress piezo actuators. However I have very low confidence in MLCCs as they most certainly will crack and the piezos from AliExpress seem to have too little travel. :/
Anyways this will be a fun little thing to play with. If someone wants a board or two, DM me

[_Wim_]

Since I also like the inbuilt superresolution-stage idea, I redrew the XY-Stage as board in KiCAD and ordered a few. Mine are based on either MLCCs or those shitty AliExpress piezo actuators. However I have very low confidence in MLCCs as they most certainly will crack and the piezos from AliExpress seem to have too little travel. :/
Anyways this will be a fun little thing to play with. If someone wants a board or two, DM me

Never thought about using MLCCCs, but indeed also a piezoelectric material. Will try to measure some this weekend and check if they move any further then the Chinese piezo's...

I like you pcb design! 

[JanHenrik]

Please do, I am interested in the results as well! I have chosen the 1812 package.

You can try to polarize the MLCC to archive a bigger stroke.
Heat to ~150°C -> apply the maximum rated voltage -> keep it in this condition for ~2hrs -> let it cool down -> turn off voltage.

[VGN]

JanHenrik, finally! I was waiting for your post!
Looking forward your progress! 

As you doing your own mechanical design also, would integrating a lever to increase the stroke be an option?

Thank you, yes, I though about that. There are special series of APA (amplified piezo actuators): https://www.cedrat-technologies.com/en/products/actuators/amplified-piezo-actuators.html
I'm sure, that these ones are going to be even more expensive...

Anyway, we could use this idea with that cheap 3.7um actuator to increase the displacement level this way, but there some disadvantages:
1. More backlash points (piezo-to-lever, lever-to-stage).
2. We will loose some resulting displacement due to lever flexibility. The stage design is going to be not that simple.
3. Lever based design probably will be quite huge, so we will have to say "good bye" to the reused original aluminium housing, though this is not a big trouble.

[_Wim_]

Today I tried with the following 1206 MLCC capacitor (X7R):
(https://www.digikey.com/en/products/detail/samsung-electro-mechanics/CL31B105KCHSNNE/3888405?s=N4IgTCBcDaIIxgOwDYC0SAsjV1QOwBMQBdAXyA)

I was unable to measure any movement with a 20Vpp sine wave (below the noise floor of the Keyence).  I did not try to pre-polarize it, as I could not detect any movement in its original state. Most other mlcc caps I have are 0805 or smaller. I did not have anything larger than this 1206 at hand. I also did not have any  Z5U or Y5V types at hand, these are maybe better for this application?

[_Wim_]

Ok, so I really wanted to know the movement of the MLCC, so I made a setup with the interferometer. But even with the interferometer I could not detect any movement with a 20Vpp 10Hz sine signal.

I connected the setup to the LCR meter and I correctly measure 1µF, so the capacitor is for sure connected.

I even switched the interferometer to FFT mode to see if it would see a 10Hz movement, but nothing. As I do not yet have on optical isolated table, this is really the lowest I can go (if somebody knows a good deal on a small optical table, please let me know).

The 2 peaks in the FFT you can see are the fans of the laptop spinning. Just toughing the mouse already gives a "huge" movement...

[_Wim_]

Just tried 4 Chinese 0805 10µF 10V mlcc caps in series on the interferometer, but even with those no detectable movement. Googling a bit more on the topic, I can find much info about the microphonics related to mechanical bending of an MLCC cap, but no real data about expansion due to applied voltage. I am starting to wonder if they really do this...

[VGN]

_Wim_, thanks for your tests! I'm not sure, but probably the voltage level is too low. A man here could achieve 800 nm over 100 V for with 1812 capacitor: https://dberard.com/2015/08/16/mlcc-piezo-actuators/


I want to show some my sketches for those people, who is interested in XY-stage development and piezo actuator testing. I don't think that I will start developing XY-stage right now, as I want to focus on new hardware and prepare it for beta-production. Anyway, hope this will be helpful.

If you look at the M-board schematics, you will find an AUX connector (X4). There are 7-pin of 3V3 IOs, connected to FPGA, single ground pin and two rails of power (3V3 and battery power 4V2-2V5). 7 IOs is enough to support two SPI devices + I2C bus (for reserved purposes).

Also I found a piezo actuator driver HV56020, that integrates a step-up DC-DC converter (up to 225V) and two high voltage amplifiers for driving two piezo actuators.
Datasheet: https://ru.mouser.com/datasheet/2/268/HV56020-Data-Sheet-DS20006335A-1843786.pdf
Mouser link: https://ru.mouser.com/ProductDetail/Microchip-Technology/HV56020T-V-KXX?qs=vmHwEFxEFR%252BS41dx8RulPQ%3D%3D

The only problem with this HV56020 - a poor documentation. I can't understand what kind of transformer is used, no information about it in datasheet. I couldn't find any application notes or examples, evalution boards, etc. Also, no support at the microchip forum: https://www.microchip.com/forums/m1144592.aspx

The schematics are attached:

[_Wim_]

_Wim_, thanks for your tests! I'm not sure, but probably the voltage level is too low. A man here could achieve 800 nm over 100 V for with 1812 capacitor: https://dberard.com/2015/08/16/mlcc-piezo-actuators/

That could indeed be the case. He only measured 300nm for 100V with a non-prepolarized DUT (which would be only 30nm with 10V in my case), and had a much "larger" capacitor (1812 vs 1206). This would mean the resulting movement in my case is indeed well below the noise floor of my setup, although I would have expected that something would be visible on the FFT...

But in my opinion these very small movements make MLCC caps also not a usable alternative for the purpose of enhancing camera resolution.

Also I found a piezo actuator driver HV56020, that integrates a step-up DC-DC converter (up to 225V) and two high voltage amplifiers for driving two piezo actuators.
Datasheet: https://ru.mouser.com/datasheet/2/268/HV56020-Data-Sheet-DS20006335A-1843786.pdf
Mouser link: https://ru.mouser.com/ProductDetail/Microchip-Technology/HV56020T-V-KXX?qs=vmHwEFxEFR%252BS41dx8RulPQ%3D%3D

The only problem with this HV56020 - a poor documentation. I can't understand what kind of transformer is used, no information about it in datasheet. I couldn't find any application notes or examples, evalution boards, etc. Also, no support at the microchip forum: https://www.microchip.com/forums/m1144592.aspx

That looks indeed like a very interesting chip for this purpose. For ready made boards you could have a look here also:

https://www.apexanalog.com/applications/ink-jet-printer.html

Edit: I did not look carefully enough, these boards still need a high voltage supply which makes them not useful.





 

[_Wim_]

The only problem with this HV56020 - a poor documentation. I can't understand what kind of transformer is used, no information about it in datasheet. I couldn't find any application notes or examples, evalution boards, etc. Also, no support at the microchip forum: https://www.microchip.com/forums/m1144592.aspx

The schematics are attached:

https://www.coilcraft.com/en-us/products/power/coupled-inductors/1-n-shielded-coupled/lpr/za9735/

[VGN]

https://www.coilcraft.com/en-us/products/power/coupled-inductors/1-n-shielded-coupled/lpr/za9735/

No way...   Thanks, _Wim_ . Passive parts designed exclusively for certain ICs are so annoying! BTW, this inductor is not avalible at Mouser/Digikey, though at least we know its parameters. Unfortunatelly there are still a lot of undocumented things, like the feedback resistor divider, diodes, dc-dc step-up caps, Rsht and so on. This is ridiculos to have such a useless datasheet for this quite non-standard application. The only hope is that microchip will finally release any appnote or evalution board.

[VGN]

Happy new year everyone! Some updates:

Still working on design port the to new hardware, but finally could get image from sensor with new hardware.

Also the most part of the HDL sources is avaliable on my repo: https://github.com/OVGN/OpenIRV
Vivado is not version control friendly, but many thanks to this man, that designed these special TCL scripts for painless version control of Vivado projects:
https://github.com/barbedo/vivado-git   
Frankly speaking, I was writing code for 20% of time and fighting with Vivado (IDE) the whole residue time.  
But fortunatelly looks like now me and Vivado are understanding each other!

I haven't finished design port yet, so you can find a lot of stubs, no USB, AV, HDMI outptut yet.

I'm using block design graphical feature of Vivado for easier development, but Vivado is quite massive (~45GB), so checkout PDF attachments of complete block design for a quick look.

[VGN]

Hello everyone! Finally updates...

I have redesigned the hardware architecture. Previously we had a single DMA and a huge FSM, that was processing the whole data transfer.
Now we have 3 separate modules, each one with it's own DMA:

1. VDMA - universal video outputing core. We have about 6 universal channels to stream video data to multiple consumers. For now only two channels are activated, the LCD and the OSD (on-scree-display for LCD). Remaining channels will be used for HDMI, AV and USB output.
2. DIP (digital image processing). This module includes all image processing cores, i.e. averaging, NUC (non-uniformity correction), BPR (bad pixel replacement), Histogram Equalization + AGC (automatic gain control).
3. Sensor module. This module is used only to control current sensor, i.e. feed it with bias data, commands and grab the video stream.

Why doing this? Because now the hardware design became more scalable. Now there is no need for massive HDL rework to support new sensors, we just have to replace the sensor capturing module with a new one. DIP became more universal too. Yes, it still depends on knowing the sensor active resolution, but I have an idea how to remove this dependency.

[VGN]

Also I have developed a special hardware bootloader - OIRVBOOT. This is a special separate "golden" bitstream, that boots first and then switches to the main application bitstream. This hardware bootloader allows to avoid the situations when the device bricks, due to incorrect update or something else. This bootloader for now is quite stupid, I can only stop in bootloder and continue booting the main application. But I'm going to add more features, like firmware update over SD-card, hardware selftesting, selective bitstream boot, backups and so on...

BTW, checkout attached picture with current NOR-flash memory space architecture. There is quite a lot of space to store another bitstream if you would like or keep any other data. The OIRVBOOT will let you to boot any bitstream at the alternate address if needed.

[VGN]

And one more thing, I have started developing the GUI logic. Checkout the attached video:



I decided to use an immediate mode GUI. The GUI that you see is based on Nuklear project: https://github.com/Immediate-Mode-UI/Nuklear
It is pretty simple, pure C, no external dependences, really good. But that was probably not the best choice for my application, as Nuklear do not support custom input, i.e. it is designed to use mouse and keyboard. That perhaps would't be a trouble, as I could map my 4 buttons to some standatd keyboard keys, but Nuklear doesn't allow to use keyboard to navigate  , only text input.

So, I'm still looking for a good GUI library with Apache 2.0 or MIT licenses, pure C (no C++), no external dependencies if possible.
I will be glad If you could recommend me something. Yesterday I came across a LVGL library: https://lvgl.io/, github: https://github.com/lvgl/lvgl
LVGL looks promising, worth to try. Any thoughts?

[ericb]

Hello,

First, congratulations for your work, this is simply awesome !!


REMOVED : I didn't see you asked for C gui, not C++

Apologies for the noise.

[sslupsky]

What OS are you using for the firmware?  You might look into the Zephyr project.  It is an embedded RTOS supported by the Linux Foundation.

[VGN]

What OS are you using for the firmware?  You might look into the Zephyr project.  It is an embedded RTOS supported by the Linux Foundation.

Hi! Oh...forgot to say, I'm using FreeRTOS. It is running on 100MHz 32-bit Microblaze MCU, avaliable RAM size is about 8MB.

First, congratulations for your work, this is simply awesome !!
REMOVED : I didn't see you asked for C gui, not C++

Hi, thanks! Yes, unfortunatelly ImGui, that you have recommended is C++, though there is a cimgui project, that wrappes original ImGui API: https://github.com/cimgui/cimgui

[pansku]

Looks like a very interesting project and I'm very impressed by the professional level of the design. Bookmarked for sure and seems like I might have to get one of those NV3 modules before summer 

[swwils]

Did you consider helical barrel arrangement for focus instead of the screw? I think a focusing helicoid would be doable with small nylon washers.



This massive one uses actual bearings but you get the idea.

[VGN]

Looks like a very interesting project and I'm very impressed by the professional level of the design. Bookmarked for sure and seems like I might have to get one of those NV3 modules before summer 

Hi, thanks! Consider email me if would like to become a beta-tester (details are in the first post of this thread).

Did you consider helical barrel arrangement for focus instead of the screw? I think a focusing helicoid would be doable with small nylon washers.
This massive one uses actual bearings but you get the idea.

Hi! In fact, no, I haven't considered this way of focusing. I'm not really good with optics, but I think that helical focusing mechanics are mostly needed for zooming, where large lens travel is required. But this camera isn't supposed to have zoom (except digital one). The main idea was to overcome the fixed lens focus distance range limitations (factory focus distance starts from ~3 meters). Fortunately 2mm lens travel screw-based focus is enough to cover distances from ~8cm to infinity. It is also quite cheap and reliable (almost no wear), reused quite a lot of original camera parts. Moreover I have redesigned some parts and fixed some problems with speed, vibrations and noise (will make a post later).

At the same time I understand that somebody probably have much better IR lens. And I don't see any problem to integrate any non-standard IR lens with this camera. Some custom plastic adapter is just needed (FDM printer capabilities should be enough). If you need some more information (dimentions or etc...) to design an adapter, just let me know, probably I could help.
This is the power of opensource.

[Vipitis]

As for lens mount, that is really a difficult one. As I am planning my own thermal camera project, it includes custom housing and a lens mount that supports interchangeable lenses. So no screws or sealing. Just swapping lenses in the field with rear lens caps and mount caps.

I have been thinking about something that locks securely like a PL mount, but the flange distance of what I can work with don't allow that at all. As one of the lens I am planing to use is rather large. It will have to be mounted to rails or a plate. Which makes changing lenses by turning and such more difficult.

I have seen screw mounts in a lot of product, mostly M39 and some smaller stuff among the ThermApp and Thermal Expert dongles. While those sensors are about the same size as the one you are working with, that's not the case for me (I am looking at 21mm diagonal - means I might be able to use a lot of older lenses and get them from sources like Bill). FLIR had their own mount for the higher end cameras like Txxx and Ex5. But it's a mount with electronical connecting for auto focus and saving a lens correction profile.

I am looking forward to what you come up with and might see a way to adapt that for my own plans... Or not.

[swwils]

Ah I understand, great!

I have bought the camera module (AUDI) and am ready to fabricate PCB.

Do you have a current BOM or should I take from schematic?

I plan to make a simple Ge or ZnSe plano-convex lens to use as macro for this system. Like described here:

http://uvirimaging.com/wp-content/uploads/2018/03/Diy-Thermal-Camera-Telephoto-Converter.pdf

This combined with the ready made optics housings from thorlabs (Ø1" Optics) - will make a very very decent and cost effective tool!

[VGN]

I have been thinking about something that locks securely like a PL mount, but the flange distance of what I can work with don't allow that at all. As one of the lens I am planing to use is rather large. It will have to be mounted to rails or a plate. Which makes changing lenses by turning and such more difficult.

Why not using a special flange adapter to get the proper flange distance for your large lens?

I have bought the camera module (AUDI) and am ready to fabricate PCB.
Do you have a current BOM or should I take from schematic?

I haven't finished testing hardware yet, hope to finish by the end of this month. Also this is supposed to be a kit with fully assembled PCBs + enclosure and other mechanic parts, though for the most brave and experienced of you, I could ship bare PCBs with BOMs, assembling schemes, etc...

[Spirit532]

Yesterday I came across a LVGL library: https://lvgl.io/, github: https://github.com/lvgl/lvgl
LVGL looks promising, worth to try. Any thoughts?

I highly recommend you go with LVGL. It's extremely simple to implement, can be augmented with 2D acceleration, and is fairly easily extensible.
I'm using it in a commercial project right now, no issues whatsoever.

[VGN]

I highly recommend you go with LVGL. It's extremely simple to implement, can be augmented with 2D acceleration, and is fairly easily extensible.
I'm using it in a commercial project right now, no issues whatsoever.

Hi, Spirit532! Thanks, going to try it!


Meanwhile...updates!

Ok, looks like I have DVI/HDMI working. For now three screen modes are supported: 800x600, 720p and 1080p.
I was a bit surprised with stable operation of the 1080p mode, as it is quite far away from FPGA capabilities according to specifications.
Anyway, it works perfectly over a 3 meter cable:

[ArsenioDev]

WHOA, that is AMAZING. Cannot wait to add this to my lab debug toolkits as an overhead capture camera, will help A TON, can just dump output into my many HDMI capture inputs.

[SilverSolder]

Truly impressive!

What is a good HDMI capture device?  The ones I have tried in the past all end up over-compressing the image...

[smason]

If youre plugging into a computer, you can look into an Elgato Camlink. Using OBS you could record quite well. this hard worked for me in the past.

[oPossum]

From the first post...

Supported outputs:
1. 320x240 TFT LCD
2. USB UVC (a common webcam protocol)
3. HDMI 480p/720p (probably even 1080p with Spartan-7 FPGA)

[fest]

Received an Audi 4G0980552 part. Although it is missing front window, I was assured by the seller that it's working. By pointing out that without the window, it's worthless for it's intended use, I got it for 160EUR.

Inside is surprisingly intact- no signs of water ingress. I was surprised to find a desiccant bag inside the camera. Judging by the oxidation layer that fell off from the main enclosure screws, nobody had opened it for a very long time, so it appears to have been there by design.

It does appear to be NV3 by the markings on the PCBs.

[VGN]

Received an Audi 4G0980552 part.

Hi, fest! Thanks much for sharing photos!
You have finally approved that Audi's 4G0980552 is exactly NV3.
BTW, internal condition is indeed perfect.

[fest]

Before trying to interface the sensor, I wanted to better understand the signals. I traced required signals on sensor board based on OpenIRV's M-board pinout- see attachments. Unfortunately, the NV3 core board does not have nice testpoints for CMD and BIAS signals, but I found them accessible on resistors on the other side of the board.

VGN, do you know if FPGA queries the sensor at all, before receiving a key?
Just probed the DATA_EVEN and CLK lines- there is a ~75MHz clock and some activity on data line (at least, as much as I could see with DS1052). Judging from the captures in Vue336 interface thread, DATA_EVEN comes from sensor only, so it appears the FPGA is receiving data from sensor even without any security key.

[VGN]

VGN, do you know if FPGA queries the sensor at all, before receiving a key?
Just probed the DATA_EVEN and CLK lines- there is a ~75MHz clock and some activity on data line (at least, as much as I could see with DS1052). Judging from the captures in Vue336 interface thread, DATA_EVEN comes from sensor only, so it appears the FPGA is receiving data from sensor even without any security key.

Exact clock value is 73.636MHz. There is no any encryption or any other secure communication between the camera and FPGA. If you send clock, valid command and valid bias values, you will get valid data on both data lines (even + odd).

[fest]

VGN, you mentioned the difficulty of obtaining bias values for each pixel. Looking at your HDL sources, I now understand that a bias value is sent by FPGA to sensor, for each pixel? That is quite interesting- in that case, it seems to me that there is a significant manufacturing variation for each microbolometer (that is, even more significant than the usual non-uniformity which is handled by the NUC).

Can you share the bias values for your sensor? I wonder if there is any discernible pattern in the values.

Anyway, I'm especially curious about this, because I hooked my unit to a logic analyzer, and the FPGA doesn't seem to send any bias values at all. Consequently after the per-frame command packet, I see something that resembles one image line, and then only the preamble on the data lines from sensor. I wonder if it's a security feature (I've heard about lockdown mode if FPGA receives incorrect data on NV2), or my unit is somehow damaged. There does appear to be some activity on the SPI flash lines, but I'll probably try dumping it next.

Failing that, I wonder what is the way detectors are characterized at the factory. Maybe it's just as simple as as binary search on the bias value for each pixel that yields the most uniform output of the detector.

[VGN]

VGN, you mentioned the difficulty of obtaining bias values for each pixel. Looking at your HDL sources, I now understand that a bias value is sent by FPGA to sensor, for each pixel?

Yes, that is right. FPGA sends 6-bit bias value for each pixel. (In fact it sends 7-bit, but only 6-bit are significiant).

That is quite interesting- in that case, it seems to me that there is a significant manufacturing variation for each microbolometer (that is, even more significant than the usual non-uniformity which is handled by the NUC).

This is also true. There are two ways to solve this problem. Bias data and command word can be retrieved from original board NOR flash (I have some success here). Another way - to recalculate this values over a two-point calibration.

Can you share the bias values for your sensor? I wonder if there is any discernible pattern in the values.

Follow this reply: https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/msg3254678/#msg3254678
I used fixed 0x25-0x26 bias value for all pixels.

Anyway, I'm especially curious about this, because I hooked my unit to a logic analyzer, and the FPGA doesn't seem to send any bias values at all. Consequently after the per-frame command packet, I see something that resembles one image line, and then only the preamble on the data lines from sensor. I wonder if it's a security feature (I've heard about lockdown mode if FPGA receives incorrect data on NV2), or my unit is somehow damaged. There does appear to be some activity on the SPI flash lines, but I'll probably try dumping it next.

This is ok, camera do not output any bias values at this point, it waits for some command from the control unit. Don't worry, your unit is not damaged, as you see preambles at the data lines (10101010101010). Preambles are generated by sensor, that means it is alive. As I said previously, there is no any secure protocols between sensor and FPGA. First three lines - are a pipeline garbage, it doesn't look like to be a thermal data, no correlation with scence or sensor temperature. The first image line you will get at the 4-th line output. But now you see zeros after preambles 10101010101010, as you do not send any bias at all. Also there is a huge pipeline delay (3 lines) between transfer of the bias values and pixel output. After command transfer we start sending bias values, then we will see 3 strange data lines from sensor (I call it pipeline garbage), only after that we will get valid thermal data. Checkout ISC0901_capture.v design file.

Failing that, I wonder what is the way detectors are characterized at the factory. Maybe it's just as simple as as binary search on the bias value for each pixel that yields the most uniform output of the detector.

I think you are right. There is only 64 levels of bias. We just have to take 128 pictures from two different simple blackdoby sources, find the bias values for each pixel that gives the best pixel dynamic output range. But picture will not be uniform at this step. We will have to calculate individual pixel gain value to get uniform image.
In fact I have already tried to calculate pixel gain values for fixed 0x25 bias. Quality of all videos and photos from my thermal camera in this thread is based on my own custom calibration.

[VGN]

Hello, some updates...

Finally have USB UVC working. Tested on linux, windows and even android. Sorry, no video for this time, but a photo.
I haven't pushed new commits for a long time, going to fix that in a few days. By the end of this week I'm going to finish AV (analog video), this is the last hardware subsystem, that I must check.

Also I have started working at documentation about the sensor, including pinout, waveforms, command pattern (at least the things I know and understand right now). This document will be updated in case of any new details.

[bap2703]

I think you're doing too well by yourself for people to post useful comments.
So in the end I can just say you're doing an amazing work 

[Hydron]

I agree that there is some fantastic work going on. I only wish I had more time available to look into your design in more detail - I would like to have a go at interfacing a ULIS sensor to it in place of the FLIR one, as I have a couple without working cores attached (both VGA and QVGA).

[fest]

Which ones do you have? Most of them seem to have analog output, although some of the newer ones have digital output (which, although is simpler protocol wise, is not compatible with ISC0901B0).

[VGN]

I think you're doing too well by yourself for people to post useful comments.
So in the end I can just say you're doing an amazing work 

Thanks for your support! It really helps to keep working)

I agree that there is some fantastic work going on. I only wish I had more time available to look into your design in more detail - I would like to have a go at interfacing a ULIS sensor to it in place of the FLIR one, as I have a couple without working cores attached (both VGA and QVGA).

Thanks! I think this is possible to do, but you will have to make a special adapter board for your sensors, check and tune power rails to suit ULIS power requirements and implement a HDL design for FPGA that will capture raw data and feed to the image processing pipelines. I keep in mind, while developing, that image processing cores should be able to support other sensors and resolutions.

Which ones do you have? Most of them seem to have analog output, although some of the newer ones have digital output (which, although is simpler protocol wise, is not compatible with ISC0901B0).

For this case we have 20 high performance IOs at the sensor connector of the M-board. I think this is enough to connect ADC, which can be placed on the special adapter board. Yep, this is quite tricky, but worth doing if you want to bring alive some really cool and expensive sensor.

[Hydron]

Which ones do you have? Most of them seem to have analog output, although some of the newer ones have digital output (which, although is simpler protocol wise, is not compatible with ISC0901B0).

VGA sensor has analogue output only, QVGA has digital as well (with a different package/pinout though ). Would only be using analogue option, and I have the power supply, biasing and ADC in hand, it's the HDL design and modification that would be the big job.

[VGN]

Hello! Some quick updates!

Ok, we have AV (analog video) working. For now NTSC and PAL of ITU-R BT.656 standard are supported. Color space is YCbCr 4:2:2.
Sorry no video again, as my DSLR is still on vacations...

[VGN]

Hi, everyone. I have some good and bad news.

1. The good one. I can say that 99% of hardware is tested. There are several non critical things that I should cover by tests, but all other stuff looks good and works properly. Of course, HW v1.0.0 had several stupid bugs, like incorrect pull-up resistor, but nothing more critical, I will describe that in schematics changelog.


2. Bad news. Right now, I'm at the step of ordering components to manufacture devices for you. I decided to use my own funds to produce the first batch of devices, as
I believe that this will be the most safe and comfortable way for me and you.

When I was developing this device I prefered to use newest and popular parts that can be easily ordered.
Well, this is sad, but for today there is a huge trouble with electronic components all over the world. I can't understand what's going on..., but most active parts just disapeared from the market. For example, the most critical component of the OpenIRV device - FPGA is non-stock.
The FPGA partnumber I used: XC7S50-1CSGA324C
Digikey: https://www.digikey.com/en/products/detail/xilinx-inc/XC7S50-1CSGA324C/7389940?s=N4IgTCBcDaIBoGEDsBlArABgLQEYEoHEBBAZjABYEQBdAXyA
Mouser: https://www2.mouser.com/ProductDetail/Xilinx/XC7S50-1CSGA324C?qs=%2Fha2pyFadug6lhtDd2nY6nfTIF3gPsXeo1XQxeK2CVrGltDj3T%252Bmgw%3D%3D
Avnet: https://www.avnet.com/shop/us/products/xilinx/xc7s50-1csga324c-3074457345632838168/
Farnell: https://uk.farnell.com/xilinx/xc7s50-1csga324c/fpga-spartan-7-210-i-o-csbga-324/dp/2828963
Of course, there are possible replacements:
XC7S50-1CSGA324I (industrial grade, more expensive)
XC7S50-2CSGA324C (a bit faster, more expensive)
XC7S50-2CSGA324I (faster + industrial...)
But these are not avaliable too. Mouser promises, that ~500 pcs. will be avaliable in september, but who knows...
By the way, the FPGA is not the only part, that is hard to order, there are several power supply ICs, that are non-stock too, though I can't remember any time I couldn't buy them.


3. I know that there are some people, who would like to manufacture devices themselves, as they have all parts from the BOM and just need PCBs.
I think that this is possible to do. I'm going to finish v1.1.0 very soon, order PCBs and send them for those who wish.


I'm a bit confused. Any suggestion are welcome.

[Max Planck]

After a quick search.

https://www.worldwayelec.com/pro/xilinx-inc/xc7s50-1csga324c/5987322

https://www.infinity-component.hk/product/Xilinx_XC7S50-1CSGA324C.aspx

I have no experience with those suppliers, so...?

Max

[Hydron]

Ouch, that's particularly annoying given all the work that has obviously gone into this. Unfortunately this is an industry-wide thing right now, the design I'm working on at my job right now has been a nightmare to source parts for too, especially with some automotive parts that are hard to get even in normal times (but commodity stuff like a STM32 have also proved difficult!).

As for the people who claim to have some, have never used them sorry, but I'm always suspicious of people who claim to have large stocks of stuff that nobody else can get, yet won't give you a straight up price or anything.

[horstmannhid]

Thanks for your detailed answer Vaagn. It is really disappointing that we don‘t get all parts right now. I also had a look and there seems to be no (trusted) supplier for the FPGA. That seems to be a global issue right now. Same with graphic cards for PCs and the automotive industry. Cannot wait to get a working camera.

Another question:
Do you know which material the 3D printing service used? I own a 3D resin printer and the parts beeing printed with normal resin are brittle and not very durable. PLA from a resin printer is much more durable for gears and so on.

[VGN]

After a quick search.
...
I have no experience with those suppliers, so...?

Thanks, I sent a letter to infinity-component.hk, let's see their answer. Though I'm agree with Hydron, parts availability and price at this market is toooooo good to true...

Thanks for your detailed answer Vaagn. It is really disappointing that we don‘t get all parts right now. I also had a look and there seems to be no (trusted) supplier for the FPGA. That seems to be a global issue right now. Same with graphic cards for PCs and the automotive industry. Cannot wait to get a working camera.

I'm not giving up! After a closer look I found out that XC7S50-2CSGA324C is theoretically possible to purchase on mouser in a two months, that's much better, that waiting for september. I should look through the whole BOM one more time to understand the scale of this problem.

Another question:
Do you know which material the 3D printing service used? I own a 3D resin printer and the parts beeing printed with normal resin are brittle and not very durable. PLA from a resin printer is much more durable for gears and so on.

Sure! All part are printed on Formlabs Form 2, the resin type is Formlabs Grey. After thorough UV curing the parts itself and the surface become as hard, that friction is very low and there is no wear at all. Gears are really very hard to break, though I'd not say that the are too fragile. I could probably make a crash test video of the focus gears if you really would like.

[sslupsky]

Hi, everyone. I have some good and bad news.


2. Bad news. Right now, I'm at the step of ordering components to manufacture devices for you. I decided to use my own funds to produce the first batch of devices, as
I believe that this will be the most safe and comfortable way for me and you.

When I was developing this device I prefered to use newest and popular parts that can be easily ordered.
Well, this is sad, but for today there is a huge trouble with electronic components all over the world. I can't understand what's going on..., but most active parts just disapeared from the market. For example, the most critical component of the OpenIRV device - FPGA is non-stock.

The problem is that a lot of companies in China purchased 6 - 12 months of supply because of existing and anticipation of more sanctions.

[swwils]

Automotives gave up fab slots expecting decreased demand then bought them back at large cost shortly thereafter (who wants to hold up a $90,000 car on JIT $45 of IC's). This, combined with other factors has sent ripples into supply chain worldwide - look at the price STMF32 and other stuff like FPGA! Even smaller specialist distribution places are being called to release all stock they have.

I expect at least a full year before it's stabilises.

[SilverSolder]

The supply instabilities will probably continue until the current p!ssing contest between Washington and Beijing gives way to more benign relations.

[robca]

4) Any idea of how much the kit minus the nv3 will cost?  I'm moving house at the moment, so counting the pennies, and want to make sure I put some money aside.  Even better if you could say some thing like the circuitry will cost about X and the housing/gears/internal parts should be about Y, etc.

I will provide this information as soon as I get the final BOM. But I'm going to keep the price competitive to HT-301, TE-Q1, HT A2, FLIR C2 and other "cheap" thermals, though OpenIRV is going to be more powerful and multifuctional. Also keep in mind that the batch size makes difference on the final price.

I'm sure it's one of the most asked questions, but before putting down my name as a potential buyer/beta tester, I was wondering if you are any closer to have a rough idea of the price range (very rough). Or even an interval of confidence (say, between $500 and $1,000)

I'm also trying to understand if your goal is to offer a full solution (even as a kit), that includes the thermal camera module, or if the plan is for people to source a NV3/Audi camera on their own (I see quite a few Audi cameras going for ~$500 on eBay)

I tried reading the whole thread, but I was not able to find a relatively recent mention of price or type of device (apologies if I missed it)

[VGN]

I'm sure it's one of the most asked questions, but before putting down my name as a potential buyer/beta tester, I was wondering if you are any closer to have a rough idea of the price range (very rough). Or even an interval of confidence (say, between $500 and $1,000)

Very rough price for the kit is definitely going to be under 500$.

I'm also trying to understand if your goal is to offer a full solution (even as a kit), that includes the thermal camera module, or if the plan is for people to source a NV3/Audi camera on their own (I see quite a few Audi cameras going for ~$500 on eBay)

There is no way to ship thermal camera modules, because of export/import restriction of different countries, though it possible to find NV3 camera almost all over the world. The goal is to design a kit, that is almost ready to use. This photo shows the whole set of part, that are going to be reused from original camera: https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/?action=dlattach;attach=1022356;image
All other parts (except the battery, due to air transport restrictions) will be included in the kit.

(I see quite a few Audi cameras going for ~$500 on eBay)

This is quite much, probably, because the part is new. Try to look for other partnumbers:
66549322653 or 9322653 - BMW
A2229052805 - Mercedes
4G0980552/4G0980552A - AUDI/WV

Also I don't think that ebay is the best place to find a good priced camera. Most second hand cameras even with cracked front window are alive, but you can hardly find them on ebay. I think this is because most sellers don't understand that cracked window camera can be reused.

[VGN]

Some updates.

1. Have ordered a new button's cover matrix for the beta release. Much better surface quality thanks to sandblasting finish.

2. Finishing schematics and PCB changes for v1.1.0. There is almost no critical issues with most of the boards. But I had to redesign the D-board (display). After hard tests I found an issue that damaged the power dc-dc at high input level voltages (+35V, i.e. 8S). I have modified the scheme a little and replaced some parts to increase the high voltage tolerance.

3. If everything with new D-board design is ok, I'm going to order a batch of beta release bare PCBs for customers, that alredy have most critical parts (like FPGA and etc...) and would like to assembly PCBs by their own. To make life easier, the bootloader will be have some self-testing features, like RAM memory test, video output test and so on... Details later.

[fest]

Inspired and informed by VGN, I also (partially) succeeded interfacing with this sensor.
I made a small breakout board and used my freshly assembled Glasgow to explore this interface. I also had to teach myself the required logic programming skills, but hey- that's what vacations are for

The breakout board and observations about the signal interface are available here: https://github.com/festlv/isc0901b0-breakout

Huge thanks goes to VGN, without whose help I could not succeed!

[1os]

Hi guys, it was such an interesting topic, but for some reason it stalled, I would like to continue......................

[VGN]

Hi guys, it was such an interesting topic, but for some reason it stalled, I would like to continue......................

Hello! No worry! The project is alive! I will report updates soon.

[VGN]

Hello everyone!

Sorry for this delay, some updates!

1. New hardware revision v1.1.0 is done. PCB are ordered, manufaturing time is about 1-2 weeks.
2. Good news. Pixel super resolution test is going to happen!
Microchip finally published evalution board user guide for piezo actuator driver HV56020!!
https://ww1.microchip.com/downloads/en/DeviceDoc/HV56020-User-Guide-DS20006328A.pdf
I have ordered parts, i.e. two ceramic actuators and the driver HV56020.
BTW, the ceramic actuator price is completely crazy, 125$ per single part (I need two of them for X and Y).
Anyway I'm sure this worth doing, at least this is going to be a proof of concept.
3. New LVGL based GUI is under development.

Thanks everybody for patience and support. I will do my best to ship set of bare PCBs for beta testers by the end of this month.
I want to remind that if you are lucky to have all parts from the BOM and have enough skills to assembly the boards yourself, this option is for you.
Stay sure, that I will provide support as much as I can.

[BiD]

Hello,

I think some Boson variant cores I had picked up off of eBay are potentially Autoliv NV4 based on Boson 640:

https://www.eevblog.com/forum/thermal-imaging/flir-boson-640-variant-teardown/msg3618032/#msg3618032

the housing isn't complete, there backing does not include automotive connector, only the standard 80-pin Boson connector is in the back.

I suspect it's NV4 because of strings found in the firmware:

* NV4-Release-60
* NV4-Authblock-Release-60

[VGN]

Hello everyone!

Some updates. As you see, I have catastrophically missed the deadlines...

Anyway, here is my progress:

1. PCBs. The whole set of five PCBs (Focus, Lcd, Buttons, Main and Peripheral) were reoreder last week.
Focus, lcd and button boards will be ready by 28 august. If manufaturer delivery service works on weekend, I will get the boards and make some photos.
Main + Peripheral boards will be done by 2 september.

Why so long, what happened? Well, there were some troubles with PCBs.
First, I found a stupid bug in a two days after PCBs were ordered (a month ago...) and asked manufacturer to stop. 
Second, I planned to make a memory controller redesign to increase stability and speed, and faced some unexpected troubles, that also affected PCB design too, i.e. I could not order PCBs again until solve this trouble. Details for FPGA designers: I decided to use oversampling mode of the ISERDESE2 and faced design routing errors with serializers clocking over BUFIO and BUFR. I was afraid that the pins at the bank I used will not allow me to make new memory controller work with new PCBs. But finally, I found the solution.

2. Reworked HyperRam memory controller. Now it is much more stable at high frequencies, no more calibration is needed, achieved 200MHz speed, i.e. 400MB/s of throughput. No more FPGA design routing errors. This weekend new OpenHBMC sources will be published on github.

3. Migrated to new LVGL GUI, demo project is working. Will show photos a bit later. Menu is still under development.

4. Reworked mechanical parts a little, i.e. focus gears. This is mostly needed for superresolution addon.

[VGN]

Updates.

1. Manufacture delivery service doesn't work on weekends, that's why focus, buttons and display PCBs will arrive tomorrow. Will make some photos by the evening.
2. New HyperRAM memory controller version in avaliable: https://github.com/OVGN/OpenHBMC

[VGN]

Today received the first set of boards (F+B+D). Main and Peripheral will be ready in a two days.
Some photos:

[VGN]

Nothing especially interesting, just a progress. Main and Peripheral boards arrive tomorrow)

[VGN]

Main and Peripheral boards arrived! 

[boywai]

I also want to join the test, how do I get the test board  Thanks

[VGN]

M-board is ready. Memory test at max speed is ok!

I also want to join the test, how do I get the test board  Thanks

Sovled in pm.

[VGN]

P-board is assembled. I could check power supply, consumption in different modes, charging, etc. - everything is ok! Time to make a functional tests.

[tisher]

What are the test results in the new version, does everything work?
When do you expect the first test sets?

[VGN]

What are the test results in the new version, does everything work?
When do you expect the first test sets?

Everything is working well! I think, I have functionally checked almost every single pcb trace.
I had a software bug in the memory controller, that I have already fixed and verified yesterday.

[VGN]

Hello everyone!

This is time to prepare sets of parts for shipment. I'm going to include not only PCBs, but also some other parts, that I hope will make life easier.

Here is an initial list of things, that I want to include in a single set:
1. Five bare PCBs, i.e. (M, P, D, B, F)
2. Silicone cover for buttons.
3. Some special screws and spacers for focus board.
4. A set of focus mechanic parts: focus_wheel, focus_worm, motor_mount, worm_mount, shutter_mount.
5. A NOR flash IC with preprogrammed bootloader. This bootloader will be able to do different functional tests (memory test, video output, etc.) and will help with board testing after assembly. Bootloader will also have firmware update feature.

But first, I want to find out how many sets of hardware should be to prepared.
Please, wait for email with other details tomorrow.

[LesioQ]

Some form of B.O.M. would be welcome, so it's clear what more is required for initial functionality testing, as it may be affected by parts availability...

[VGN]

Some form of B.O.M. would be welcome, so it's clear what more is required for initial functionality testing, as it may be affected by parts availability...

Sure, BOM will be published today.

[VGN]

Several minutes ago I have set emails to everyone in my list. Let me know, if I forgot anybody, in this case send me an email with title: "OpenIRV.Developer/Tester". You can find my email address in attached pictures of the very first post of this thread.

[Myrddin]

I was researching the idea of a XY displacement enhanced thermal camera when I found this thread, very impressive VGN! I have the feeling I am a few minutes late but if you still need beta-testers let me know, I would absolutely join a crowdfunding and I am happy to help with beta-testing, documentation,.. if needed. Already ordered a NV3 (broken window, crossing fingers...) on ebay     

Also the nudge to finally register to the forum and not only reading all the interesting stuff that's going on here...

[NVTech]

If anyone orders a NV3 camera and would like a used, but defect free window, let me know.
I’ve got quite a few lying around atm.

[VGN]

I was researching the idea of a XY displacement enhanced thermal camera when I found this thread, very impressive VGN!

Hi, thanks)

I have the feeling I am a few minutes late but if you still need beta-testers let me know, I would absolutely join a crowdfunding and I am happy to help with beta-testing, documentation,.. if needed. Already ordered a NV3 (broken window, crossing fingers...) on ebay     

You are not late, consider email me with title: "OpenIRV.Developer/Tester". You can find my email address in attached pictures of the very first post of this thread.

Also the nudge to finally register to the forum and not only reading all the interesting stuff that's going on here...

Welcome)


Added BOMs v1.1.0: https://github.com/OVGN/OpenIRV/tree/master/docs/BOM/BOM_v1.1.0

[Myrddin]

I have got a NV3 spare part for 120 Euro, unfortunately the lens is missing. Before I return it, does someone needs the part?

I have the feeling the original lens is quite good, so it does not make sense trying to exchange it with e.g. cheap laser lenses or other parts. High quality lenses are more expensive than a complete new unit...

[NVTech]

Is it just the primary lens or just the protective flat lens at the front of the unit?

[Myrddin]

Is it just the primary lens or just the protective flat lens at the front of the unit?

The primary lens, I thought just the protective flat window at the front, but it turned out the lens is missing as well.

[NVTech]

That’s disappointing, Sounds like they may have tried to short change ya there.
They may have removed the lens and tried to sell on the camera as if it were complete.

[tisher]

VGN, we can use any FPGA XC7S50 or XC7S25 core with 324?

[VGN]

VGN, we can use any FPGA XC7S50 or XC7S25 core with 324?

XC7S25 cannot be used. Mostly because IO BANK35 and BANK16 are NC in XC7S25. Also design is too huge to fit in XC7S25.

That's why, only XC7S50:

• XC7S50-1CSGA324C (commertial grade, slowest, cheapest) - I actually use this one.
• XC7S50-1CSGA324I (industrial grade, more expensive)
• XC7S50-2CSGA324C (a bit faster, more expensive)
• XC7S50-2CSGA324I (faster + industrial...)

[tisher]

It's possible to find this core in automotiv parts on scrapyard? 

[ArsenioDev]

Yes, you can absolutely find it in scrapyards, though it's not super common.

[VGN]

Updates!

My last order for focus parts printing failed. No worry, that was a small test order. Long story short, that was my bad, I allowed manufacturer to use another type of resin, that looked to be the same. As a result, the mechanical parts' surface is not strong/stiff enough as it used to be, that caused too  high friction. With previous type of resin the parts were like a glass, that's why gears were working well, without any significial friction and wear. Both worms at the photo looks identical, but left one is not working at all.

At that moment I realized, that I strongly need a SLA to control of the whole process. That's why last week I spent looking for a 3d printer. Fortunatelly I could find secondhand form 2 in a perfect condition. I lauched a test run, to see how it works. If everything is ok, I will start printing all parts.

P.S. look how SLA heats up while working!

[1os]

Hi guys, but the topic is very delayed, it would have been very relevant a year ago. Now the Chinese are tearing apart thermal imagers with low prices and very good quality, I have been doing a thermal imager for a long time and now American flir has receded into the background, it is very difficult to sell American flir.

[1os]

I would like to know what the price will be for boards without a sensor, at least approximately so that it would be interesting to watch this topic approximately 100 200 300 or 400 US dollars If you answer this will be a big plus it interests everyone thanks

[ArsenioDev]

Hi guys, but the topic is very delayed, it would have been very relevant a year ago. Now the Chinese are tearing apart thermal imagers with low prices and very good quality, I have been doing a thermal imager for a long time and now American flir has receded into the background, it is very difficult to sell American flir.

W/R/T this, not designed so much as a turnkey core, more like a multi-sensor testbed and experimentation platform to tinker with the information pathing and data processing of a FPA output through image generation. For those of us on the R&D side of the world, this is a killer app and would help a ton for being able to quickly test and develop with new FPAs for validation purposes.

[VGN]

I would like to know what the price will be for boards without a sensor, at least approximately so that it would be interesting to watch this topic approximately 100 200 300 or 400 US dollars If you answer this will be a big plus it interests everyone thanks

If you are asking about fully assembled PCBs, I cannot say exact price right now, nobody knows when parts will be avaliable and how the price will change.
But if we are talking about prices before this global silicon shortage, I think that was possible to keep fully assembled KIT price around 450$ or even lower.
Also take in account that the more KITs manufactured => the lower is the price for a single set.

[VGN]

Updates.

Test prints were successfull. I have fixed some small bugs. Now parts perfectly fit each other.
I'm starting "mass" production)

BTW, I have redesigned the magnets horling ring, as prevoius one was too weak. Also added shutter mount spacer.

[VGN]

Some progress.

Hope to have almost all parts to printed by the end of this week. Part look very good, no print fails at all.

[RO]

As a FDM printer owner I still am amazed at the resolution of these resin machines, it is looking very good.

How is the production going for you VGN?

Still very impressed with the progress so far, especially considering the difficult times.

[VGN]

Hello, RO! Hi everyone!

How is the production going for you VGN?

I have been pretty overworked last tree weeks. But no worry, I'm continuing.

1. I had some trobles with scaling of the printed parts. The SLA printer repeatability is quite high, but the scale factor way wary much, depending on the size and geometry of the part. The best solution was to print same type parts simultaneously and adjusting the XY scale. I could achieve +/- 0.05mm accuracy. All plastic parts were printed. I will make photos by the end of this week.

2. I have ordered additional sets of PCBs, as 10 PCBs (-2 for me), that I have now is not enough. The should arrive next tuesday.
Also quite a lot of people asked me to add a small adapter board for programmer, it was ordered too and will arrive next week.

3. SPI NOR flash ICs should arrive this week. I already have and adapter to preprogram the initial bootloader with selftest firmware.

4. Silicone buttons are also in production, hope to finish them by next week.

P.S. I don't know why, but these worms look like bacteriophages...

[littleshrimp]

Hi,VGN
When I use 0x25 as bias, I get the this image. Is my FPA damaged?

[Fraser]

Depends what you mean by “damaged”. Your microbolomter has 4 dead rows and a lot of column noise. The dead rows are hardware faults, the column and row noise is dealt with in software. 4 dead rows would be unacceptable on a lab grade camera but dead pixel correction could cope with it in less demanding roles. As a rule, dead pixels are common on microbolometers but whole dead columns or rows are not. Such would often result in a microbolometer QA test failure. That microbolometer may have been acceptable for its original application in a vehicle however.

Fraser

[drops206]

Hello everyone.  At the outset, I would like to say how amazed I am when I look at the work of VGN.  The idea to use the system from a car to make a thermal imaging camera is already impressive, but I did not even think about such image quality and refreshing time.  If I found such a product in a store, it would not have occurred to me that it was all DYI.  It's nice to see people so involved in their project.  Some time ago I became the unfortunate owner of FLIR One Pro which turned out to be damaged (I bought it on ebay, the money was recovered and there was no contact with the seller).  I had a plan to use a LEPTON sensor with a simple adapter, but a friend told me about this project.  There shouldn't be a bigger problem with the camera itself (we have a lot of car parts), but what about the rest?

[redbook]

I came across this interesting article about the ISC1406L sensor ("micron" teardown). 

FLIR Boson IR Camera Core

>> PDF

[gsumner]

Hi, I've started looking at a Tau2 640x512 sensor in hope of interfacing with this project. The sensor interface sounds like it might be identical to the ISC0901. I've attached an image what I think the sensor is doing. I guess the last two signals are pixel valid/enable like in the HDL? I can't see a CMD signal however.

VGN, could the 3 lines of pipeline garbage you note in ISC0901_capture.v be values that reflect/help with the row/column noise? Could they be Unexposed pixels? My camera reliably has empty, data, empty for the three lines and I feel like there's a chance it's not garbage.

I've linked a sigrok trace of a (empty) frame https://drive.google.com/file/d/1uf3K03b041-RHY4dU8iX2qOJ5yBbgxgw/view?usp=sharing in case someone wants to look. It shows the bias -> pixel delay and 3 lines of garbage. Frame period is 1/30sec (despite being a 9Hz camera)

[ArsenioDev]

Greg! Glad to see you in thread finally.
Yeah, most decent FPAs will not be 9Hz, rather are just FPGA gateware locks, not on the imager side.

[VGN]

Hi,VGN
When I use 0x25 as bias, I get the this image. Is my FPA damaged?

Hello!
No, I don't think that your FPA is damaged, I saw something like that when I was bringing up my FPA cores.
As Fraser said, looks like there are 4 dead rows, but probably not, because probably these 4 rows require some special bias value to start working.
I call pixel dead only when it outputs only 0 or 1 regardless to any bias value in the whole range 0 - 63.
Note that real bias value for ISC0901 is 6 bit:
bias[6] - no effect
bias[5:0] - actual bias range 0-63

Also there are some fields in command, that should be set proreply. BTW, I found a way to extract original bias and command pattern values from the original firmware, just full dump of the original SPI flash is needed. I think this is the only solution for the beginning, to make the sensor work. Later I'm sure it is possible to find an algorithm that will brutforce the 64 bias values along with some fields in command to find get the best image output. This procedure can be done at the step of the "two point" calibration, when each pixel gain factor is calculated. I will add a SPI flash dumping feature in bootloader image, that will help to dump the original flash image using OpenIRV hardware. Details a bit later.

[VGN]

There shouldn't be a bigger problem with the camera itself (we have a lot of car parts), but what about the rest?

Hi, thanks a lot. If you are asking about OpenIRV hardware parts - yep, this a great trouble, almost no any FPGAs or some other parts avaliable in stock all over the world, due to global chip shortage. We can do nothing with it, just wait when this hell finishes or we all find a way to deal with it. The initial plan was to ship fully assembled boards, but even people from USA cannot buy part, so I decided to start shipping soon only bare PCBs for those, who was lucky to get parts and have enough skills to assemble the PCBs.

I came across this interesting article about the ISC1406L sensor ("micron" teardown). 

You can easily find articles about other sensors, including ISC0901: https://www.systemplus.fr/wp-content/uploads/2017/06/SP17330_Autoliv_Night_Vision_FLIR_microbolometer_Sample_System_Plus_Consulting.pdf
Unfortunatelly, some extemely interesting images are blured. For the pixel shifting part of this project I would like to see the SEM photos of the pixels, especially the pixel area size, the gaps between pixels. Hope that 18um piezo actuator displacement will be enough to make a full cycle shift within a single pixel in both X and Y directions.

[VGN]

Hi, I've started looking at a Tau2 640x512 sensor in hope of interfacing with this project. The sensor interface sounds like it might be identical to the ISC0901. I've attached an image what I think the sensor is doing. I guess the last two signals are pixel valid/enable like in the HDL? I can't see a CMD signal however.

Hi, Gleg!

I think the last two pins at your image are enable and cmd, there is no pixel valid strobe from sensor. Try to probe both residue lines again, there should be a command line. The waveforms from your dump really looks identical to ISC0901. Please, checkout attached .vcd dump. You can use gtkwave software to open the .vcd file.

VGN, could the 3 lines of pipeline garbage you note in ISC0901_capture.v be values that reflect/help with the row/column noise? Could they be Unexposed pixels? My camera reliably has empty, data, empty for the three lines and I feel like there's a chance it's not garbage.

In fact I still have no idea what these 3 lines contain. I couldn't find any correlation of the content of the first 3 lines and the actual scene.
On the other hand, there could be a garbage in registers after power up, but no idea why it continues outputing this garbage for all other frames.

[MegaVolt]

What beautiful work and a well done project. Gave me an aesthetic pleasure looking at it!!!

Too bad I saw the project late... I had already ordered the HT-301 ( I already regretted a lot

[littleshrimp]

Thank you for your reminder. I found 2 sets of 336*256*2 bytes of data in the original SPI flash file. The data is similar to 0xA0xx, 0x9Fxx. I used these data to generate this image. Are they bias data?

[littleshrimp]

Except for the 4 "dead rows", the main problem now is that it can only be used to detect objects with relatively high temperature, and the image at about 40 degrees Celsius will be submerged in noise.

[gsumner]

Greg! Glad to see you in thread finally.

Thanks for the welcome, I was hoping to show up right before xmas and get a present!

I think the last two pins at your image are enable and cmd, there is no pixel valid strobe from sensor.

You're right thanks, I can see a command once per frame on Pin 10 and 11 is enable.

I'm not sure if the nv3 lens needs it, but I've made a prototype (tinfoil + tape) of a calibration chessboard. The different emissivities look like they might give good enough contrast to get a calibration from opencv (https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html). The test image shows a warm reflection in the aluminium foil but I think you could connect a CC supply and reverse the contrast when the tape was hot. I guess you could expose it onto a PCB and put a heater trace on the back too? Then we'd be able undistort images on the FPGA.

[VGN]

Thank you for your reminder. I found 2 sets of 336*256*2 bytes of data in the original SPI flash file. The data is similar to 0xA0xx, 0x9Fxx. I used these data to generate this image. Are they bias data? (Attachment Link)

Unfortunately no, these are gain values. These are in fact almost useless tables, as I already have an algorithm that can recalculate each pixel gain using two uniform images with a temperature difference about 30-40 degree, that is called two point calibration. This is quite easy to do. The quality of thermal images, that you saw in this thread are based on this calibration. Also bad pixels are detected and marked for pixel replacement pipeline.

Except for the 4 "dead rows", the main problem now is that it can only be used to detect objects with relatively high temperature, and the image at about 40 degrees Celsius will be submerged in noise.

This is ok, as soon as you have proper command pattern, bias values, you will be able to recalculate the pixel gain. Finally after offset correction, that is periodically done by internal shutter, your camera image quality will meet my level.


Some updated.

Some boards arrived. These are the PCBs, that actually will be shipped. There is no routing or any other changes at all, this is the same v1.1.0, but with gold finish.
I decided to replace immersion silver finish with immersion gold, as gold has longer shelf time and there is no tarnishing issue.
Waiting for Main and Peripheral PCBs that should arrive in a 2-3 days + small debugging adapter board.

[gsumner]

I'm intending to make a flex PCB to adapt the Tau2 through-hole pinout in my previous pictures to the Molex 52991-0408 to match the ISC0901B0 pinout.

VGN, do you think it would be reasonable to have a common bitstream between the two FPAs if I use an ID resistor on the adaptor? Maybe a sensor ID in general might avoid frying an FPA?

BTW, I think X1 and X2 are mixed up between the PCB and documentation.

[gsumner]

Hi, here's a picture of what I'm thinking of making. It's my second go at a PCB, anyone have suggestions? It's ~26mmx30mm. It'll fold down 180 and present the same interface as the ISC0901B0.

I'm assuming the OpenIRV M board is the same size as the autoliv board so I don't think it will fit in the Tau case.

[VGN]

I'm intending to make a flex PCB to adapt the Tau2 through-hole pinout in my previous pictures to the Molex 52991-0408 to match the ISC0901B0 pinout.

VGN, do you think it would be reasonable to have a common bitstream between the two FPAs if I use an ID resistor on the adaptor? Maybe a sensor ID in general might avoid frying an FPA?

I haven't decided yet if it is reasonable to keep a common bitstream. Probably not, as this going to be a waste of logic. Also some FPAs will require another level of bias, core or vccio voltage, that can be tuned by feedback resistor only. If some FPAs are very common, I think that common bitstream could be used.

BTW, I think X1 and X2 are mixed up between the PCB and documentation.

This is because I haven't updated schematics yet. In the repo now is v1.0.0, while new boards are v1.1.0. I will upload new schematics at this week.


BTW, main and peripheral boards finally arrived.

[VGN]

Hi, here's a picture of what I'm thinking of making. It's my second go at a PCB, anyone have suggestions? It's ~26mmx30mm. It'll fold down 180 and present the same interface as the ISC0901B0.

Good job!
1. I think you could make connector's (2.54 pitch) via annular ring size much smaller to increase clearance, that will decrease the PCB cost.
2. Again, still not sure about id resistors. This is a matter of discussion. How about i2c or even better, 1-wire smallest and cheapest eeprom to keep the ID?

I'm assuming the OpenIRV M board is the same size as the autoliv board so I don't think it will fit in the Tau case.

Right, I think a 3d printed adapter can solve this problem.

[gsumner]

Right, attempt two! this one should allow an I2C or UNI/O EEPROM. I left the I2C option in because they go up to 2MBit. I think with some compression you could store something interesting like the bias values with the sensor. With UNI/O it will only use one IO.

I think we can only rely on VCC_SYS being present in the bootloader and I've assumed the IO voltage will start off at 2.5v. Hopefully SENSOR_IO_0 could sense the pull-up to stay compatible with an FPA that needs every pin(?). VGN, would you be interested in using this in the bootloader to ID other sensors?

Switching IO voltage after the bootloader could be a problem though. What if you passed VCCIO_FPGA through? These EEPROMs can do 1.8v to 5.5v, I could get rid of the regulator and they'd follow the IO voltage.

[VGN]

Hello, I have finally uploaded schematics v1.1.0 to the repo: https://github.com/OVGN/OpenIRV/tree/master/docs/schematics

Right, attempt two! this one should allow an I2C or UNI/O EEPROM. I left the I2C option in because they go up to 2MBit. I think with some compression you could store something interesting like the bias values with the sensor. With UNI/O it will only use one IO.

ISC0901B0 requires 84KB for a single bias frame. As I know, original NV3 has 4 table to cover wide camera temperature working range, i.e. we should store 84x4 = 336KB of bias values, i.e. we need 4mbit eeprom. But as I know, you can hardly find 1-Wire, UNI/O or even I2C eeprom of such capacity. For higher resolution sensors, we probably will require even more. Also loading bias table from I2C eeprom will be slower than from QSPIx4@65MHZ, that impacts on statup time.

On the other hand, I think that some kind of attached board identification is needed. That's why I want to suggest having 1-wire smallest eeprom, that will just keep some identification data. In fact, we already have a pull-up resistor R6 at the pin 19 of the X2 connector of M-board. This resistor is planned to be used to identify that original ISC0901B0 sensor board was attached (as there is GND on this pin). We also can use this pin 19 as a communication line for 1-Wire eeprom of the addon board.

I think we can only rely on VCC_SYS being present in the bootloader and I've assumed the IO voltage will start off at 2.5v. Hopefully SENSOR_IO_0 could sense the pull-up to stay compatible with an FPA that needs every pin(?). VGN, would you be interested in using this in the bootloader to ID other sensors?

We can also rely on VCCIO_FPGA and VCCIO_FPGA+SENSOR (controlled over Q3 mosfet). Bootloader can switch VCCIO_FPGA+SENSOR power on, that will power a 1-wire ID eeprom, which will be able to communicate with FPGA over pin 19 of X2 connector, that already have a pull-up resistor R6. Though I think that this resistor should be a bit stronger, i.e. probably 4.7K will be fine for 1-wire. How about AT21CS or 11LCxxx EEPROMs ?

Switching IO voltage after the bootloader could be a problem though. What if you passed VCCIO_FPGA through? These EEPROMs can do 1.8v to 5.5v, I could get rid of the regulator and they'd follow the IO voltage.

This is already done) Just look though the schematics again, BTW I have uploaded lastest v1.1.0 schematics. VCCIO_FPGA+SENSOR is the same VCCIO_FPGA, controlled over Q3 mosfet.

[gsumner]

ISC0901B0 requires 84KB for a single bias frame. As I know, original NV3 has 4 table to cover wide camera temperature working range, i.e. we should store 84x4 = 336KB of bias values, i.e. we need 4mbit eeprom. But as I know, you can hardly find 1-Wire, UNI/O or even I2C eeprom of such capacity. For higher resolution sensors, we probably will require even more. Also loading bias table from I2C eeprom will be slower than from QSPIx4@65MHZ, that impacts on statup time.

I was thinking that because the bias is 6bit it's only really 64KB per bias array. A lot of the data should be similar(?) and compress well. But I didn't realise there were at least 4! And slow too, I take your points. I wonder how compressible they are though.

We can also rely on VCCIO_FPGA and VCCIO_FPGA+SENSOR (controlled over Q3 mosfet). Bootloader can switch VCCIO_FPGA+SENSOR power on, that will power a 1-wire ID eeprom, which will be able to communicate with FPGA over pin 19 of X2 connector, that already have a pull-up resistor R6. Though I think that this resistor should be a bit stronger, i.e. probably 4.7K will be fine for 1-wire. How about AT21CS or 11LCxxx EEPROMs ?

I didn't read the schematic very carefully sorry. I assumed the regulators were being digitally adjusted with PWM or something. So I was thinking we had to decide which voltages to program and make sure to not enable anything in the bootloader... 

I'll add a 1-Wire EEPROM like you've suggested. Would it be fine for the bootloader to detect presence by trying to communicate the whole time pin 19 isn't pulled down?

[littleshrimp]

There are 4 groups of 336*256*9byte data in FLASH, are they encrypted bias data?

[horstmannhid]

Hi Vaagn,

do you think it is possible to order the stencils for most of the PCB‘s ?
I think otherwise it is a pain to assemble them.

[PlainName]

Ignore that comment, please - it was ill-conceived 

[VGN]

There are 4 groups of 336*256*9byte data in FLASH, are they encrypted bias data?

Not only bias, also complex offset table, some kind of gain, etc. There 4 tables to cover wide temperature ranges. Original tables are quite complex, as I understand they have precalculated offset values to minimize requirement of frequent shutter calibration. I decided not to use this model. Bias, gain and offset table is enough for hand device.

Hi Vaagn,
do you think it is possible to order the stencils for most of the PCB‘s ?
I think otherwise it is a pain to assemble them.

Hello. I haven't decided yet. At least I'm going to opensource stencil gerbers. But not sure if it going to be included in KIT. On the other hand, you are right, assembling M-board and P-board is a pain, though I have assembled them without stencils.


Concernig project news, will make an update on this weekend.

[horstmannhid]

Hello. I haven't decided yet. At least I'm going to opensource stencil gerbers. But not sure if it going to be included in KIT. On the other hand, you are right, assembling M-board and P-board is a pain, though I have assembled them without stencils.

Hi, would be great if you could just release the stencil gerbers. So evererybody who needs this can order it by themselve. It wont be that expensive. I want to use my reflow oven for assembly. Thanks for your answer. I‘m already excited about the news coming about the new development status.

[VGN]

Hi, would be great if you could just release the stencil gerbers. So evererybody who needs this can order it by themselve. It wont be that expensive. I want to use my reflow oven for assembly. Thanks for your answer. I‘m already excited about the news coming about the new development status.

Stencils are uploaded: https://github.com/OVGN/OpenIRV/tree/master/HW/gerbers/stencils

[VGN]

Added JTAG passive adapter board schematics: https://github.com/OVGN/OpenIRV/tree/master/docs/schematics

[ckambiselis]

Hi,
Are you still "taking orders" for this, I have an E4 for some time and a NV3 coming in a few days and would be interested.

Oscar

[VGN]

I'm sorry for disappearing, I could not get out from the shock and stupor last two weeks.

I hadn't found a spoiler tool in message edit mode, that's why I hided the beginning of this post in a the attached txt. Read txt first.


Nevertheless:

1. I'm not going to stop working at this project.

2. I need some time to find a way to ship KITs.

3. In fact I already have everything to continue software development. Some progress with software will be published within 2 weeks.

4. I don't want to let people down, who like this project and already bought cameras to get them work.
Stay sure that in the worst case I will just opensource everything that I have.

5. I got a lot of email last two weeks and will send reply within this week.

6. I barely had a chance to manufacture and test JTAG adapter.

[alcs2000au]

Thanks for the update! Feeling with you and hoping everything works out and you can continue your awesome work! Looking forward to it 

[sslupsky]

Please stay safe.

[VGN]

Hello everyone.
Some news.

1. Right now there is no realistic way to send PCBs to backers and receive payments.
That's why I decided to share all gerbers + PCB stack details, i.e. eveything you need to purchase PCB manufaturing.
I will not commit gerbers to github for now. If you would like to get gerbers, just email me. You can find my email in the attached pictures in the first post of this thread.

2. I'm was pretty overloaded by daily job, that's why software development was paused...
I hope to have much more free time by the end of this week. At least I must finish bootloader firmware, that will help to bring-up the hardware.


P.S. I'm really sorry for this situation, but this is the best thing I can do now for you and this project.
Thanks for understanding.

Best regards

[RO]

Please do not apologise after already putting in this amount of effort, and sharing and open sourcing so much information already.

Your work is awesome!

I can not deny, I to would like to have my hands on it sooner then later, and every update message I rush to the board hoping to finally have something for my expensive paperweight autoliv... But let's be honest, that's because I lack most if not all the skills you have, and can't even get the damn thing booted properly.

I am mostly glad you haven't abandoned the project (but didn't think you would) grateful for your updates and looking forward to your product

[gsumner]

Hi All, A little while ago I received my flex boards back, the holes are unfortunately a little undersized but I don't really want to pay for round 3! They will still press-fit over the pins, I just have to hope they're functionally correct now.

Does anyone else have a Tau 2? I intend to open up the design if the adaptor works correctly and am wondering if there's interest.

[VGN]

Hello everyone!

I have already shared gerbers to people who asked me and will continue to do it.

One more VERY important thing. As you know, to bring a single PCB set cost down you should order as huge volumes as possible. That's why:
1. I won't mind if you combine orders to keep the price as low a possible. You could probably cooperate within a country, or continent, ha-ha))
2. Feel free to use this thread to collect orders.

Best regards!

[gsumner]

Hi, does anyone from the UK want to combine orders? I'm looking at Euro Circuits and for the M-Board (8 layer) alone it's ~£310 (5x) or ~£350 (10x). Quite a bit unless we have a few people.

I don't think it'd make sense for me to order for EU people, customs would cost everyone more. If there are lots of EU orders I'd be happy to join in on an EU order, it'd probably still work out better if there's good volume.

[svgurus]

Does anyone else have a Tau 2?

I have tau1 640 which has broken ribbon cable connector on green board, does your solution fit tau1?)
If anyone from Russia/Belarus has spare autoliv nv3 lens, I'd buy it or trade for my nv2 lens or lens + some case details. I also have some interest in nv1 lens, especially if its not m34 or is machineable down to m30-32

[Chanc3]

I'm in the UK and possibly interested... Just need more spare time and funds for this project.

[gsumner]

I have tau1 640 which has broken ribbon cable connector on green board, does your solution fit tau1?)

I'm not sure sorry, I've never seen a Tau 1. I think you'd have to replicate the investigation I did earlier in the thread. I don't even know if the adaptor works with the Tau 2 yet though I guess as long as the power lines are correct the rest could be sorted in the fpga? I think this pdf: https://flir.netx.net/file/asset/12412/original/attachment suggests the Tau 2 is an incremental upgrade over Tau 1. It does mention higher frame rates but all the features look like they could be fpga improvements to me. Anyone else know?

I'm in the UK and possibly interested... Just need more spare time and funds for this project.

This project will help you see how quickly you're burning through money though

[NVTech]

I’m from the U.K.
If we can get the prices down I’d be interested in buying 2 kits.

[OC]

Hi all.

I'm from the UK and would also be interested in 1 or 2 kits.

Cheers

[Kurt-Guo]

Hi, guys.
I'm in China and I just asked about the price of the 8 layers M and P board.
If I can combine the M and P board Gerber files together. I can get both boards for about 1900RMB（50Pcs in 48 hours, mat black with 1μm gold).
And if only ship the M and P board, I think it is OK to just ship it with tracked airmail so the shipping cost should be relatively low.
Let me know if there is anyone interested in getting the board together.

[LesioQ]

If I remember correctly, shipping from China is usually cheap .... ? 
So how it boils down to single kit numbers, shipped, estimate USD/EUR/PayPal ?

[Denis_K]

Hello!
I join those who want and plan to make OPenIRV camera.

I bought one NV3 camera for the project, but it turned out to be corroded inside, including the sensor, although it looked good on the outside.

I also bought an NV1 camera (BMW 6654696982401) to study it.
As it turned out, it has a video DAC adv7127 and control is via the LIN bus. Unfortunately, I don’t have a control unit from it, so I don’t have any data on the exchange protocol, if someone has robbed volume data on the LIN bus, then please share)
Inside the NV1 camera, an FPGA from Altera EP1C12 is used, and a CPLD from Xilinx xc2c128, which is in the same JTAG circuit with ALTERA and is probably used to fill it with a configuration from a parallel CFI compatible FLASH.

The hardware of the NV1 looks exactly like the PCB from the PatnfindIR camera, which does not require activation and produces video immediately after being turned on. My idea is to copy the contents of flash memory from PathfindIR to FLASH Autoliv NV1 to end up with a camera that outputs video without any external commands)

But there are 2 problems:

1. I have not yet been able to figure out how to read data from flash memory via JTAG (I could only read CPLD). A rough solution is to unsolder FLASH and read / write with a programmer.

2. I don't have a PatnfindIR camera. If any of you have, could you get the data from the flash memory?


The sensor from the first generation camera does not have a voltage of 3.3 volts, but has 5 volts. I attached my observations on the pins in the photo.

[MIKUTA]

Hi, Denis_K
 I have a control unit and camera removed from the same vehicle. I know that night vision is activated when certain data is sent to a specific address, but I feel that I also need a switch on the driver's side to see that data. I will let you know if I make any progress.
" alt="" class="bbc_img" />

[newex]

Could you share a CPLD dump?

[Denis_K]

Could you share a CPLD dump?

[Citizen]

Excellent, I'm just blown away by the work done. I myself have implemented several simpler projects (PCB design, C programming, case, cooling) - I know it's very difficult!
The zoom capabilities are amazing!
Impressed by the beauty of PCB design, I myself am engaged in PCB design, but I don’t know how to be so beautiful. Previously I worked with Altium and Eagle, but now I do many projects on EasyPCB - I like that you can easily share the project as on github, as well as order partial mounting of SMD components on one layer - this saves a lot of time during assembly.

[Citizen]

Denis_K, we can cooperate with you, the following advantages:
- logistics chains are well established between our countries;
- in connection with the current terrible events, several of my colleagues in Armenia (next to the author)));
- I have access to the European auto parts market, NV3 is much cheaper there than in our countries.

[Citizen]

I have small experience with UAVs, and absolutely no any experience with FPV, FPV-cameras, interfaces, etc... Are you still flying on NTSC or PAL? 
That's why I have some question to you and community:

1. What do you think about this addon? Is it worth doing?
2. What about total weight and dimentions?
3. What kind of interfaces FPV pilots use nowadays? Is AV enough? What is the output format (NTSC/PAL/or...)?
4. What else is reasonable to add to the board?

I have experience.
Focusing - not needed for flying, but needed for ground (for example, a robot for surveying communications) drones.
Need Ingress Protection (IP).

The following interfaces are required:
- AV (PAL25, NTSC30) - yes, this is relevant, there is no delay;
- USB (for Raspberry Pi);
- miniHDMI (720p60);
- PWM / UART / PPM - switching palette, focus (if implemented) and calibration (on, off, automatic).
For the broad masses of users, PWM / PPM is relevant.
In my projects, I like to implement such interactions on UART:
- 2 bytes for synchronization;
- information byte settings flags (palette, calibration, autofocus);
- information byte 255 gradations of focus;
- the last byte is the modulo sum of two information bytes.

Power supply 5v (required) or 8-60v (optional).

Interchangeable back panel, only with required interfaces. For example coaxial sma/mcx for AV and 4 pin power/uart.

[Denis_K]

In my projects, I like to implement such interactions on UART:
- 2 bytes for synchronization;
- information byte settings flags (palette, calibration, autofocus);
- information byte 255 gradations of focus;
- the last byte is the modulo sum of two information bytes.

I think, for unification, you can repeat the uart protocol, such as flir Photon, tau, etc .. (there is a detailed open description of the commands, the file is attached). At least get away with it.

[Citizen]

I think, for unification, you can repeat the uart protocol, such as flir Photon, tau, etc .. (there is a detailed open description of the commands, the file is attached). At least get away with it.

Yes good idea

[MIKUTA]

 I don't have the technical knowledge to understand exactly what is going on, but I heard that the camera can be activated by following the URL below.
https://github.com/pavelmalik/BMWCanBridge
https://www.bmwcustomretrofit.com/navigation-retrofits/BMW-Night-Vision-System-Activator

[Citizen]

I don't have the technical knowledge to understand exactly what is going on, but I heard that the camera can be activated by following the URL below.
https://github.com/pavelmalik/BMWCanBridge
https://www.bmwcustomretrofit.com/navigation-retrofits/BMW-Night-Vision-System-Activator

I know similar projects, but they use the original video processing unit from the car. The activator is already working with this block.

[LesioQ]

I tried to find this info in the OpenIRV docs, but couldn't : what kind of LCD display should be used for this project ?
I mean the model / type that needs no further firmware modifications.

[rockwell]

Should one need a new camera, I have one to let. contact by PM.

[VGN]

Hello!

Have added datasheets for LCD and focus motor: https://github.com/OVGN/OpenIRV/tree/master/docs/BOM/BOM_v1.1.0
You can easily find both. Note that there are a lot of manufacturers that make same 6mm geared DC motors.
Use provided motor datasheet as a reference. The key parameters are diameter (6mm), gear ration (~1:26), shaft geometry and voltage (nominal 3.0V).

[LesioQ]

Thank You. That clarifies things further. And how is Your progress with bootloader part ?
I guess FPGA needs to be programmed ISP, so will bootloader be involved ?

[NVTech]

Hi VGN,

I'm just looking for a little clarification on the SLA parts.
In the "housing" folder there are two Bottom parts, one is labelled H13, the other H7. I've scanned through the thread and cant find an exact reference to these numbers, though you must forgive me for using the Ctrl + F function for this as there is a huge amount of info to look through here.
Do I need to choose between the two, or do I need both for one full camera assembly?
I'm about to print these and would like to save a bit of resin if possible by only printing the one(s) I need.

Also, I'd like to thank you for spending so much time developing this amazing project to such a high standard.

[1os]

Greetings. I've been following your thread for a long time. I see that you are very well versed in the device of thermal imaging cameras. I had a problem with my thermal imager, it shows, but there are some artifacts in the picture. I want to send you some pictures. I don't know what to do with this access to the thermal imager settings I have. this is a flir tau 2 640 thermal imager but most likely it is the same meaning as the thermal imager with which you understand. Can you help me somehow. in the thermal imager, the flash memory from another similar thermal imager was replaced, what needs to be changed in the settings so that it works correctly. everything is fine with the lens.

[1os]

.

[1os]

.

[1os]

I try firmware from different cameras, I get this video

[1os]

these are not broken pixels, this is something else I don’t know, pixels will be visible on the sides on a clean image.

[Denis_K]

these are not broken pixels, this is something else I don’t know, pixels will be visible on the sides on a clean image.

Perhaps this is due to the calibration tables that are in flash memory.

[k2tikhin]

You are probably or:
1. having issues with MX flash memory and a part of gain data has been lost
2. using sensor or flash from a different camera

in both cases try to calibrate via Flir GUI, there is an option to do it there. If it's not working - more work needed with flash dump raws.

[k2tikhin]

as of BMW NV1 (Autoliv) issue - it can work as a stand alone camera, but you have to
1.program XA2C128 via JTAG with another dump - there are just 2 versions of dumps I've been faced with - Autoliv camera and Flir PF
2.read dump from intel flash via JTAG and with Cyclone pins "help". Xilinx chip should be blanc to not interfere. This 28F640j flash has 8Mb. Gain data stored at
1A0000-200000
2c0000-360000
3e0000-440000
but you need to read the whole flash
3.once you have it done, you need to read dump of 28F640J from PathFindIR camera. create a new file using gains (addresses above) from autoliv camera and everything else from PF, winhex is good for that.
4. as far as autoliv, PF and Flir Photon have the same Samtec connector wiyh the same pinout, you can use Flir GUI to calibrate the camera.
JTAG is better, but takes time. in case you are brave enough - you can desolder 28F640J from the board (be careful with the compound not to damage the board and other BGAs around - Cyclone and MT48V8M16).
To read flash via JTAG you have to have FPGA/Flash connection matrix.
everything above written is based on my experience and it's working.

[NVTech]

Hi, are you still able to get SEM images of a sensor if I send you a full camera?
I have a few spare NV3 cameras lying around that could be put to good use...

[zrq]

Maybe, if there is no issue shipping to Switzerland and I can find a way decapping the sensor (no experience. So if you can do this, sending SEM ready samples will be more helpful).

[tomasis]

Hi, are you still able to get SEM images of a sensor if I send you a full camera?
I have a few spare NV3 cameras lying around that could be put to good use...

Denis_K, we can cooperate with you, the following advantages:
- logistics chains are well established between our countries;
- in connection with the current terrible events, several of my colleagues in Armenia (next to the author)));
- I have access to the European auto parts market, NV3 is much cheaper there than in our countries.

what is approx price of such part? do you have part number? Mostly BMW?

[NVTech]

No problem at all, I take it the sensor is effectively unusable afterwards due to the pre processing for SEM imaging?

I got a few cameras free of charge so if feel this would be a worthwhile contribution to the project.

[NVTech]

Send me a DM if you’d like to buy?

[zrq]

Sounds good. It would be nice if you can remove the package of the sensor, which will certainly make it unusable by venting it and likely introducing dust onto the die. Please PM for address.

[NVTech]

Does anyone know if the following STEP files are available, I've had a look in the GitHub repository and can't locate them?

Magnets holding ring
Shutter mount spacer
Updated motor mount

[OC]

I believe that the software files for the application running on the MicroBlaze are also missing from the GitHub page. I cannot find a built file of the application either.

I am close to having a fully built M-board so this is a bit of a showstopper.

[LesioQ]

Try getting to VGN by e-mail. There's quite a bunch of us who put money on finishing the prototype cameras based on his commitment and declarations. 
Fingers crossed !

[LesioQ]

Does anyone know where this project got stuck ?
Are we bricked ?

[OC]

Hi all,

Just posting in case people have a stash of parts that have been out of stock for a while.

I am missing the following ICs to complete my M-board. If anyone has them and is willing to sell them, so that this project can be rejuvenated!

TPS61041DBVR
TPS7A4901DGNR
TPS7A9001DSKR
TPS62821DLCR

Or know of any drop in replacements!

[dmendesf]

TPS61041DBVR - 4k pieces at Mouser
TPS7A4901DGNR - digikey has similar part BD00FA1WEFJ-E2
TPS7A9001DSKR - Texas has 6k pieces
TPS62821DLCR - digikey has 11 pieces

Finish it and show it to us

[dmendesf]

Hello!

Have added datasheets for LCD and focus motor: https://github.com/OVGN/OpenIRV/tree/master/docs/BOM/BOM_v1.1.0
You can easily find both. Note that there are a lot of manufacturers that make same 6mm geared DC motors.
Use provided motor datasheet as a reference. The key parameters are diameter (6mm), gear ration (~1:26), shaft geometry and voltage (nominal 3.0V).
[/quotwe

Hello!

Have added datasheets for LCD and focus motor: https://github.com/OVGN/OpenIRV/tree/master/docs/BOM/BOM_v1.1.0
You can easily find both. Note that there are a lot of manufacturers that make same 6mm geared DC motors.
Use provided motor datasheet as a reference. The key parameters are diameter (6mm), gear ration (~1:26), shaft geometry and voltage (nominal 3.0V).

Hi VGN, hope everything is ok. Chippagedon is less serious now. Will you publish the spice , only the binaries or have something else in mind?
Daniel

[ArsenioDev]

Really hoping this hasn't died

I'm in the process of getting captures of several other microbolometers and sourcing currently manufactured ASi units to allow for custom builds without requiring the donation of an entire NV3, ironically not that hard to get em, bigger headache will be the lenses.

[Fraser]

VGN was last logged into this forum back In December 2022. I fear that this project has stalled until he has the time to revisit it.
This is why I do not financially invest heavily in a project that is still being developed. Many such projects fall by the wayside when the developers circumstances or motivations change. I am equally guilty of such but tend not to publish my projects so it is no problem.

I was interested in what was achieved by VGN and it is so close to being a complete project, but close is not enough when it comes to creating a useable project such as this. With the advent of cheaper imaging cores and complete cameras coming out of Asia, such projects can become redundant or non competitive before they are ever finished. The trials of being a product developer.

Fraser

[ArsenioDev]

True, My NV2 at least will get me rollin along with my matched set of VGA cores I have the ICDs on. In any case, what the baseline we got here will be a useful future reference.

[horstmannhid]

I think some of us have got a complete set of the boards. And we also have got the manufacturing data of the boards.

I tried to load the existing software into vivado with the OIRVBOOT.tcl file but I have some problems running the RTL analysis. But I think that's only some sort of creating a new project and load the existing files.

The main problem may be the missing microblaze software. Does anyone have got a completly assembled board and running some code on it?
A working display only showing the thermal image would be great in a first step. Would be so sad if nothing further happens with this project.
I have a lot of experience programming microcontrollers in C but I have not much knowledge with FPGA's.
Main problem are the costs of all the components. Otherwise I will just try it.

[ArsenioDev]

Now that I've obtained the FPA and have some FPGAs on order, I'm working with some coworkers to re-do this from the baselines that exist here and take it on our own path as it appears VGN has gone dark.

[horstmannhid]

Now that I've obtained the FPA and have some FPGAs on order, I'm working with some coworkers to re-do this from the baselines that exist here and take it on our own path as it appears VGN has gone dark.

That sounds awesome.

I now ordered a development board (Arty S7-50) with exactly the same FPGA. In the first step I will try to build a simple camera with the MLX90640 and a TFT-Display. If I make any progress with that I'll continue assembling the boards from VGN.

A nice tutorial suitable to our project can be found here:
https://community.element14.com/technologies/fpga-group/b/blog/posts/arty-s7-50-vga-thermal-imaging-camera

[Mach]

Hello guys,

i know this is not the core business here, but, by any chance - anyone tried to dissasemble the original Audi night vision control unit, and extract firmware?

Im trying to retrofit it into a non-compatible car - Q7 system to a Q5. Everything works, only the image transfer does not occur (but it throws out an error after transmission attempt - also calibration not possible after tool is found).

one of two likely applies:
1) Virtual cockpit cluster does not have HW or SW on it (unlikely, as it has all adaptations, codings possible, HMI parameters are identical,...)
2) NV Control unit, against vehicle gateway or cluster requires a handshake (chassis ID/VIN; VIN mask - this was the case on the non-encrypted modules) -> More believable, as there are many NV control units for different audis, and even for other in the VAG concern: vw's, Bentleys, Porsche,....

i do have the original firmware, but its encrypted (likely AES, no compression). There is a chance, that on the board in the chipset the decrypted data could be downloaded. For a known fact, flashing procedure on car opens protocols for decryption and checksum actions.

if anyone can help, let me know.
Sorry for asking non-topic related question, but im running out of ideas

regards, M

[railrun]

You mean something like this?

https://www.eevblog.com/forum/thermal-imaging/autoliv-nv3-unlock/

[horstmannhid]

Made some progress with the development board and just the small MLX90640 sensor. But it’s a long way to go. And it’s really complicated programming FPGA’s and dealing with the Xilinx software. Maybe I‘ll start to interface the camera with an adapter board in a first step.

[daedolon]

horstmannhid, you should probably know about these two projects. I feel like they've done a lot of the work for you.

https://github.com/randomplum/autoliv_nv3_uvc
This one uses a Gowin FPGA and an FX2 to convert the sensor's output to a USB webcam. He hasn't done calibration yet, and there's no shutter control, but otherwise it works. He used the high level python-based Amaranth HDL to program the FPGA so it should be easier to work with. Obviously it would be easier to use the same FPGA he's used, but you can also more easily port it to other FPGAs: https://amaranth-lang.org/docs/amaranth/latest/platform.html

https://github.com/festlv/isc0901b0-breakout
This is the project the USB webcam converter is based on. This one's "just" a breakout board that supplies power. It does have unidirectional shutter control. However in his board VGN used a H-bridge driver to give bidirectional shutter control (which also gave him a driver for the focus motor, as it's a dual H-bridge driver). The shutter I have definitely needs bidirectional control to return fully to the open position, but it's somewhat damaged so I'm not certain this applies to undamaged shutters.

[horstmannhid]

Hi daedolon,

thanks for your answer. I didn't know about the first project. Maybe it's also helpful.

I knew about the second link. I just wanted to order the breakout board. Just refitted the connector to a one that is possible to order at Digikey. But I think it doesn't make sense to assemble all that because it's also not cheap. I have got no final plan yet. I think I will assemble the original boards from Vaagn as I already have got them and try to get it running with the existing code. I will share the Digikey part lists here.

I succesfully opened the Vivado project in Vivado 2020.2 and ported it to the actual version 2023.2. Don't know yet if it all works with the new version but I created two pdf's of the last block design from this project.
The microblaze has to initialize at least the BRAM's for the color palettes and of course the VDMA. What I don't understand right know is how the color palettes (axis_palette_lut) should look like. But I think that's a thing that I'll find out. A lot of example code how to initialize the IP's can be found here: https://github.com/Xilinx/embeddedsw.github.io/blob/master/axivdma/examples/vdma_api.c
My own test with the small MLX90640 also had a VDMA and works with two frame buffers. But its only a small test project which has got no final stage and is very slow because a lot is running on the microblaze. So nothing to really share 

And of course the microblaze has to create an OSD (on screen display) image with 0xFFFF for every tft pixel (320x240) to just show the thermal image. When you want to show a menu it is possible to show it as an overlay. That's how the "lcd_ctrl.v" is designed.

That's so far what I found out. Hopefully I'll get it running.

[horstmannhid]

https://github.com/festlv/isc0901b0-breakout
This is the project the USB webcam converter is based on. This one's "just" a breakout board that supplies power. It does have unidirectional shutter control. However in his board VGN used a H-bridge driver to give bidirectional shutter control (which also gave him a driver for the focus motor, as it's a dual H-bridge driver). The shutter I have definitely needs bidirectional control to return fully to the open position, but it's somewhat damaged so I'm not certain this applies to undamaged shutters.

Sorry forgot to say. Your shutter is damaged. I have got an unused NV3 camera and the shutter is returning itself with a spring.

[horstmannhid]

We have to use Vivado 2020.2. The Hyperram module isn't supported by the newer versions of Vivado but that doesn't matter.

Here are the BOM's for the parts at digikey. I chose some replacement parts which are available to order. Almost everything is in stock. I will not assemble everything of the P-Board because it's expensive and not needed in the first step.

OpenIRV_P-Board https://www.digikey.de/de/mylists/list/Y2DV4BHYMK

OpenIRV_M-Board https://www.digikey.de/de/mylists/list/E8V9XF2ZES

OpenIRV_F-Board https://www.digikey.de/de/mylists/list/4EW9TYYM5X

OpenIRV_D-Board https://www.digikey.de/de/mylists/list/U7877XPDF7

OpenIRV_B-Board https://www.digikey.de/de/mylists/list/49EETGAR8X

[daedolon]

Thanks for all that. I'm a software guy by trade, plenty of firmware experience, but none with FPGAs. I'm hoping to find the time to stick my head into these HDLs and see what I can do with them.

Thanks for the tip on the shutter too, that's what I suspected. I'll see if I can hook a return spring on it externally.

[horstmannhid]

The hardware in Vivado is finished and should hopefully work. I already synthesized it and created a bitstream in Vivado and exported everything to Vitis. In Vitis it's "only" some C code writing for initializing everything.

Takes some time for me to build everything. There are still some components missing. Cables etc. I will post an additional BOM with all additional parts needed.

[horstmannhid]

OpenIRV_additional_parts https://www.digikey.de/de/mylists/list/TFRJUL1OOY

Display: https://www.ebay.com/itm/291850983213?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=C6-gh6yGR66&sssrc=2047675&ssuid=&widget_ver=artemis&media=COPY

[isuzulake]

TKS,It's very cool. I wish you sucessful for it. 
I like it.

[sslupsky]

Wonder if it would be worthwhile to consider porting the FPGA to an open source platform such as LiteX?
https://github.com/enjoy-digital/litex

[horstmannhid]

Some news. PCB's are all assembled and the FPGA is programmable. I also got the Microblaze running and the LED is blinking.

Problem now is how to initialize the VDMA which is programmed completly by Vaagn. Any help would be great :-)
A simple "Hello World" on the LCD would be great in a first step.