Author Topic: OpenIRV. ISC0901B0 (Autoliv NV3, FLIR E4/5/6/8) based opensource thermal camera  (Read 149424 times)

0 Members and 1 Guest are viewing this topic.

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hello everyone!

This topic is about my attempt to develop an opensource high performance and multifunctional thermal camera based on a ISC0901 sensor.

I use thermal imaging to diagnostic malfunctions of different devices under development or repair. Five years ago I have developed a small camera based on FLIR Lepton 2 80x60. It works pretty well, but in fact this is a toy for me, and I was looking for something better. Unfortunately FLIR EX series didn't suit for me too for many reasons, including poor fixed focus lens, a bit inconvenient form factor, price, etc...
I'm also a supporter of opensource projects, but all of them are still based on FLIR Lepton 2/3 products or some other tiny sensors.


Once an Avtoliv NV3 camera module occured in my hands. I've disassembled it and found out that this camera module has a great potential and perfectly suits my new project, because:
1. It is based on very popular ISC0901 336x256 sensor. Popular, because NV3 camera is used on tons of different vehicles till now, you can find this spare part very easily. FLIR EX and ETS320 are based on the same sensor.
2. It is an external shutter based camera. External shutter allows to significially improve the image quality.
3. It has a pretty fast (15mm in dia) telescopic lens! This lens is much better that FLIR EX lens.
4. Pretty nice form factor (60x62mm)


I decided to develop a camera that will be multifuntional and can be used in many different ways and applications:
1. Hand tool, like FLIR EX
2. Stand tool, like ETS320
3. Thermal camera for robotics and UAVs, like FLIR Vue


Supported outputs:
1. 320x240 TFT LCD
2. USB UVC (a common webcam protocol)
3. HDMI 480p/720p (probably even 1080p with Spartan-7 FPGA)
4. AV (NTSC/PAL and many other formats if necessary)
5. Micro SD card (SPI mode + SDIO x4 in future)
6. 3-pin configurable 5V tolerant GPIO to support any low speed interface (UART, SPI, I2C, 1-Wire, PWM, PPM, SBUS...)


Power:
1. Standard Li-ion 18350 battery. Battery life is about 3-3.5 hours (with 1100ma battery), depends on number of active outputs, LCD backlight, etc.
(I'm still working on increasing this value, but I think that reaching 4 hours is a good result.)
2. USB charge (<500ma). USB power is enough for continues camera run with all outputs enabled.
3. External power over aux connector 5.0-35.0V (i.e. up to 8S battery) for UAV mode.


Some features:
1. Adjustable motorized focus system (details later).
2. Supports frame rate up to 60fps (details later).
3. Two standard 1/4" metal mounts on the top and bottom.

More photos/videos and detailed stories of each side of this device will be later too.



Repository:

Follow this project on GitHub: https://github.com/OVGN/OpenIRV

Some sources are not available right now, but I have plans to publish them a bit later.



For beta-testers, developers and backers:

Consider to email me if you would like to be a beta-tester, developer or just a backer.
It would be great if you keep mail header like this: "OpenIRV.Developer/Tester" or "OpenIRV.Backer" or etc...
I promis not to publish your email anywhere else and do not send mails, that are not relevant to OpenIRV project.
I always reply letters. You can easily find my email in attached images ;)
« Last Edit: December 06, 2020, 01:15:15 pm by VGN »
 

Offline Cat

  • Regular Contributor
  • *
  • Posts: 96
  • Country: de
Awesome, this looks very promising :-+
I'm looking forward to learn more about your project, especially how you interfaced the µbolometer and implemented UVC.
Didn't think it would be possible to get a motorized focus inside the tiny housing.
Keep up the good work!
On the Internet, nobody knows you're a cat.
 
The following users thanked this post: VGN, NVTech

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
What a superb project, well executed and very professional in its appearance   :clap:  :-+

As I own one of these cameras I will await further details with GREAT interest  :-+

Fraser
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 
The following users thanked this post: VGN

Offline zrq

  • Frequent Contributor
  • **
  • Posts: 343
  • Country: 00
Quite interested in the interface with the sensor and the reverse engineering. Can't wait to see the code! (and hopefully writeup)
 
The following users thanked this post: VGN

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Awesome, this looks very promising :-+
I'm looking forward to learn more about your project, especially how you interfaced the µbolometer and implemented UVC.
Didn't think it would be possible to get a motorized focus inside the tiny housing.
Keep up the good work!
Thank you! Looking ahead, I used FX2LP for USB. FPGA is streaming data over 8-bit bus. As a start point I used this Cypress project: https://community.cypress.com/docs/DOC-14406. More details further... With regard to the focus system, first I didn't believe too)

What a superb project, well executed and very professional in its appearance   :clap:  :-+
As I own one of these cameras I will await further details with GREAT interest  :-+
Wow, many thanks! :)  As you are owner of one of these cameras, I hope for your help with testing ;)

Quite interested in the interface with the sensor and the reverse engineering. Can't wait to see the code! (and hopefully writeup)
Hi! The interface with the sensor is, obviously, the most tricky moment. In fact, there are a lot of things that I still don't understand, but I hope for community help.
 

Offline conmega

  • Contributor
  • Posts: 23
  • Country: us
I also have an NV3 and was working on reverse engineering efforts to interface with the original circuitry.
I just got some boards I made for interfacing to the NV2 going out to some friends to help with development/testing. Although its mostly used as a test platform not such a professional looking project as yours. I look forward to your progress and wish you luck!
 

Offline tmbinc

  • Frequent Contributor
  • **
  • Posts: 253
Very cool project!

Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?

Can't wait to see more details!
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I also have an NV3 and was working on reverse engineering efforts to interface with the original circuitry.
I just got some boards I made for interfacing to the NV2 going out to some friends to help with development/testing. Although its mostly used as a test platform not such a professional looking project as yours. I look forward to your progress and wish you luck!
Hi! I think we could colaborate for better result. I don't have NV2, as I know, this is a previous generation camera, it has an internal shutter (behind the lens). Sure you know, there is a great work about NV2: https://debugmo.de/2018/12/autoliv-nv2-teardown/

Very cool project!
Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?
Can't wait to see more details!
Hello! Thanks much! Yes, you are right. I refused using original circuitry in my project, because it may not survive, when the front window is broken. Also original electronics limits some opportunities.
« Last Edit: July 13, 2020, 10:44:12 am by VGN »
 

Offline Max Planck

  • Regular Contributor
  • *
  • Posts: 97
  • Country: pl
Bought some time ago an NV3, but never had time to work on it.

Thus, I'm really impressed by the project.
Are you using just the FPA or also some parts of the NV3?

Max
 
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?
Thus, I'm really impressed by the project.
Are you using just the FPA or also some parts of the NV3?
Thanks you! Ok, I thinks this will be the start point of the story.

I have three cameras, two of which have cracked front windows (btw cost me under 100$ for each). I didn't expect that condition of the most critical parts, i.e. sensor and lens will be perfect. In fact all three sensors are alive with amount of bad pixes <1%.

1023338-0

As you know, the front window is very fragile. As soon as it catches a stone from the road and cracks, your camera will soon stop working, because of water. Some water damage pictures:
1022336-11022340-2 1022344-3 1022348-41022352-5

Nevertheless the sensor itself in located in the center of the camera module, and incomming water do not reach it, as the water level do not rise above the bottom edge of the broken window. Perhaps that's why both sensors of drowned cameras are working. Of course you should be carefull, there is no 100% garanthy that sensor will work, but at the same time the probability is quite high. The shutter motor can also be easily replaced if necessary.

You can also see a lot of small dents at the surface of the windows. But the surface of all lens are in perfect condition. I think this is because lens is located very deep inside of the enclosure and also protected by broken window. Anyway that gives a chance to bring alive very poor cameras.


Here you can see parts that I have reused from the original camera:
1022356-6

The front window and cap is optional. I don't really use them, because it is an additional obstacle for IR radiation and the device itself is not water or dust protected. It is designed mostly for home and limited outdoor usage.
« Last Edit: July 14, 2020, 06:27:23 pm by VGN »
 
The following users thanked this post: zrq

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
So if it is just the casing, lens, FFC flag and Microbolomter PCB that is used in your design, presumably this project could be adapted to the earlier NV2 model ?

You have basically built a thermal imaging camera from scratch which is a significant achievement when you do not have the datasheet for the microbolometer or its recommended bias voltage values. Building the back end video processing package and firmware is a further very impressive achievement. You have skills  :-+ Much respect for you  :clap:

Fraser
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
So if it is just the casing, lens, FFC flag and Microbolomter PCB that is used in your design, presumably this project could be adapted to the earlier NV2 model ?
Yes, you are right. I believe it can be adapted to any camera, that needs deep raw image processing.

You have basically built a thermal imaging camera from scratch which is a significant achievement when you do not have the datasheet for the microbolometer or its recommended bias voltage values. Building the back end video processing package and firmware is a further very impressive achievement. You have skills  :-+ Much respect for you  :clap:
Thanks much! Bias values were picked by hand, I'm still looking for a way to calculate this values properly. Do you have any knowleges about that? After tons of experiments I found out that this is 6-bit value per pixel and could properly feed them to FPA core.
 

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
The bias values are normally provided by the fabricator of the microbolometer and are unique to each one. The user of the microbolometer then uses these values as a start point from which they are fine tuned for the cameras particular application, temperature range etc. The bias voltages can be fine tuned for best NeTD. Sadly I do not have any bias voltage values for FLIR microbolometers.

Fraser
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 

Offline Treehouseman

  • Supporter
  • ****
  • Posts: 58
I've been thinking that it could be possible to extract the bias values from the flash chips, I would imagine it's just a semi standardized mapping that if you compare 2 camera's flash images and see what the differences is it may stand out. It would then be a task to see how they're storing it but with the NV2 given that we have the code from it's softcore, the softcore may load in the values, and that could tell us how they're stored. Especially if they keep the same structure they used for debugging where you could also send it values manually for initial calibration.

While we don't currently have the program from the nv3 it's possible it uses the same data encoding. On top of that we could also see how it's used in a Flir E4 or a Flir E40 (NV3 and NV2 respectively) and see if it's the same thing. I would assume they just drop the same table of values in.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Well, there are a lot of things to tell about, so for simplicity I'm going to move step by step from outside to inside.

The whole enclosure consists of four parts:
1. Top (where the display is mounted)
2. Middle
3. Bottom
4. Original black casing

All parts were developed for CNC machining, but I decided to try SLA 3D printing for prototype. Of course, the quality is not super, there are some wracks at the surface and edges. Anyway I'm sure CNC machining with further casting plastic in silicone molds will give much better quality.

1022886-0

The bottom:

1022858-1

The middle:

1022862-2

The metal mounts is nothing but a 1/4" to 3/8" thread adapter. Fitting the adapter from the inside prevents possiblity of unwanted unscrew:

1022866-31022870-41022874-5

The top:

1022878-6

Finally I had to increase the lens hole a little to prevent field of view narrowing, because previously I had to raise this black case up to place the focus mechanics.
There is a round boarder that helps to make a neat cutout. This cutout will still allow you to fit the front window if you need it, but maybe you will also have to increase the hole of the window cap too.

1022882-7
« Last Edit: July 14, 2020, 12:17:32 am by VGN »
 
The following users thanked this post: Fraser, Max Planck

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
Great to read this thread as it gradually reveals the work you have done. Loving this  :-+

Fraser
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 

Offline Max Planck

  • Regular Contributor
  • *
  • Posts: 97
  • Country: pl
First technical question.
I know, it is maybe to early at this stage, but have you taken into account the heating of your camera enclosure during operation (electronics, battery, external heat sources) and its temperature changes on camera (FPA) operation?

Max
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Great to read this thread as it gradually reveals the work you have done. Loving this  :-+
Thanks a lot!

First technical question.
I know, it is maybe to early at this stage, but have you taken into account the heating of your camera enclosure during operation (electronics, battery, external heat sources) and its temperature changes on camera (FPA) operation?
Yes, it's taken into account. As you know, all FPA pixel are different, they all have different response to the same amount of IR radiation. So to solve this problem and make the image uniform we should determine a gain and an offset parameter for each pixel. Gain parameter changes with the camera temperature quite slowly, whereas the pixel offset changes at a temperature difference of ~0.2°C. To compensate this temperature drift FFC (Flat-field correction) procedure is initiated. We use external shutter to close the front of the lens and assuming that shutter surface is uniform, we perform recalculation of offset parameter, improving the image. You can run FFC manually any time you want to improve the image quality or enable automatic mode, when firmware monitors enclosure temperature by integrated temperature ICs and initiates FFC periodically.
« Last Edit: July 14, 2020, 04:17:05 pm by VGN »
 
The following users thanked this post: Max Planck

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
The display module.

1023374-01023378-11023386-21023390-3

This is the biggest LCD that I could find to fit in 60x62mm dimentions. Also manufacturer promises long term supply for this part (10 years since 2018).

Brass thread inserts are glued into plastic by cyanoacrylate. That looks ugly, that's why plastic molding is preferable for manufacturing.

Buttons cover is made of platinum cured silicone with black pigment. It's pretty reliable and looks good. As you see the button's plunger is quite small, so I decided to place additional 0.3mm plastic plunger that will allow to increase button's press area. Plastic plungers were cutted by hand and looks also ugly, but laser cut will fix that.

There is also a small bug, the buttons board should have additional mounting hole in the center. Will be fixed in future revision.

Few words about electronics. The CLC filter reduces power noise from AUX port. DC-DC converts 5.0V-35.0V to 4.5V. Unfortunately, there was no enough space to fit this regulator on the mother board, that's why it is here. All inductors are shielded to reduce possible EMI noise. LCD backlight is powered by constant current regulation IC (not PWM) also to minimize the power noise level.

1023382-4



Feel free to ask questions and criticize the design, that may help to make it better.
« Last Edit: July 14, 2020, 07:38:00 pm by VGN »
 

Offline newex

  • Contributor
  • Posts: 23
  • Country: 00
The amount of work you have done is amazing.
I think it makes no sense to criticize the design, because we do not see the schematic diagram, but it looks great and professional.
 

Offline Vipitis

  • Frequent Contributor
  • **
  • Posts: 880
  • Country: de
  • aspiring thermal photography enthusiast
I am more than impressed by the effort so far. Wish I had the knowledge, experience and tools to mount such a project. But really shows what can be done... and motivates me to move on with my project (in 14 days.)
 

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
I am loving this thread. I very much look forward to seeing future updates. Your camera project looks very professional and not unlike what I have seen in R&D prototyping. Very professional in all respects.

I might add that even with the skills to create the mechanical and electronic design, that is not enough for such a project, as I well know. The fact that you also have the skills required to write the code for the hardware is very impressive :-+

Fraser
« Last Edit: July 14, 2020, 11:01:08 pm by Fraser »
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 

Offline Max Planck

  • Regular Contributor
  • *
  • Posts: 97
  • Country: pl
Yes, it's taken into account. As you know, all FPA pixel are different, they all have different response to the same amount of IR radiation. So to solve this problem and make the image uniform we should determine a gain and an offset parameter for each pixel. Gain parameter changes with the camera temperature quite slowly, whereas the pixel offset changes at a temperature difference of ~0.2°C. To compensate this temperature drift FFC (Flat-field correction) procedure is initiated. We use external shutter to close the front of the lens and assuming that shutter surface is uniform, we perform recalculation of offset parameter, improving the image. You can run FFC manually any time you want to improve the image quality or enable automatic mode, when firmware monitors enclosure temperature by integrated temperature ICs and initiates FFC periodically.
Yes, you have two things, NUC and thermal dirft caused by several factors. I was asking about the latter. You are correcting it using a shutter. My question was about the dynamics of that process, i.e. how often the shutter has to be activated to keep a decent image quality/temperature measurement accuracy.

Knowing about some of the challenges you were and are facing, I'm impressed.

Max 
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
The amount of work you have done is amazing.
I think it makes no sense to criticize the design, because we do not see the schematic diagram, but it looks great and professional.
Thank you! Well, yes, I realize that this hard to criticize the design without the schematics, but I mean, maybe you have some global ideas that do not require schemes currently. The problem is that I'm going to make quite a lot of changes for the next hardware revision and I think that it would be better if I publish the very last schematics. I believe we will have enough time to discuss and make changes to design if necessary before I send new files for manufacturing.

I am more than impressed by the effort so far. Wish I had the knowledge, experience and tools to mount such a project. But really shows what can be done... and motivates me to move on with my project (in 14 days.)
Thanks much! I wish you good luck with your project!

I am loving this thread.
...
Many thanks, Fraser!

Yes, you have two things, NUC and thermal dirft caused by several factors. I was asking about the latter. You are correcting it using a shutter. My question was about the dynamics of that process, i.e. how often the shutter has to be activated to keep a decent image quality/temperature measurement accuracy.
Knowing about some of the challenges you were and are facing, I'm impressed.
Thank you, Max! Basically one should perform FFC every time when camera's temperature changes for a some threshold value up or down. Also there is no need to perform a FFC if temperature is stable, the image quality will not get worse in time. I can't exactly say how often the shutter has to be activated, as it depends on the environment temperature, how fast the camera heats up and the temperature change threshold value. I don't think that this a point of worry, until you use it on UAV, because FFC causes an image freeze for a short period of time (~1 sec). There are two ways to mitigate this problem. First - you can initiate FFC yourself, second - the firmware can display a countdown before FFC to help the pilot get ready.
« Last Edit: July 15, 2020, 05:20:00 pm by VGN »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Peripheral board (P-board):

1024282-01024302-1

Some descriptions:

1024286-21024290-3

This board do not process image at all, but provides output path to different interfaces, manages battery power, powerpath logic and some other stuff. D-board (display) and B-board (buttons) are connected to it through 12-pin and 5-pin JST connectors. You can also see a 6-pin JTAG header that is used for FPGA debug on the M-board (main), which I will show futher.

1024298-41024294-5

Some device safety features:
  • All IOs at HDMI, USB and 7-pin AUX port are ESD protected.
  • USB 5V power line has overcurrent (>500ma) and overvoltage (up to 20V) protection.
  • AV output has an output buffer with short-to-ground and short-to-battery (up to 18V) protection.
  • AUX external power line has high side reverse polarity protection. (high side protection type is probably not the best choice...)
  • Three GPIOs at the 7-pin AUX port that goes directly to FPGA are protected by 5V tolerant bidirectional buffers.
  • Battery reverse polarity protection. Magic smoke will not appear even if you install battery wrong for a continues period of time.
  • Battery is also protected by a special charge/discharge voltage/current protection IC like in factory protected Li-ion batteries. I refused using protected Li-ions, as their length can vary too much from manufacturer to manufacturer, you simply won't be able to insert the battery into holder, meanwhile dimentions of unprotected batteries are much more predictable.

Other features:

  • RTC provides timestamps for SD card filesystem.
  • Temperature sensor.
  • Power reset button.
« Last Edit: July 16, 2020, 12:12:50 am by VGN »
 
The following users thanked this post: joe-c

Offline joe-c

  • Frequent Contributor
  • **
  • Posts: 351
  • Country: de
    • Joe-c.de
its a really awesome project.
I am very interested.

currently, I work on an open-source solution too, but its more an interface of an existing Camera, not a completely new device.

I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.

anyway... really nice. Thanks for show it.
best regards
Freeware Thermal Analysis Software: ThermoVision_Joe-C
Some Thermal cameras: Kameras
 

Offline Ismsanmar

  • Regular Contributor
  • *
  • Posts: 53
  • Country: es
VGN, incredible work. Another one here interested in the possibility of adapt your solution for the NV2 ISC0601 sensor. I'm another one with one of those lying around.


I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.

The problem with that camera is that only goes at 15fps on line detection mode
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
its a really awesome project.
currently, I work on an open-source solution too, but its more an interface of an existing Camera, not a completely new device.
I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.
Thank you!
No there is no visual camera, but I thought about that. You know, I held Flir EX in my hands once. And I can't say that MSX technology impressed me a lot. MSX cannot help when FPA's resolution is low itself or when you cannot change the lens focus. Also image alignment is not possible at short distances. Looks like MSX brings more engineering headache than benefits. Are there a lot of people using MSX? Maybe you have some arguments that can change my mind?

VGN, incredible work. Another one here interested in the possibility of adapt your solution for the NV2 ISC0601 sensor. I'm another one with one of those lying around.
Thanks!
Yes, I think it's possible to adapt this camera for NV2 sensor. But I don't have it, and know completely nothig about ISC0601 sensor. If you have this camera, you could investigate the pinout, how is it powered, how many digital IOs should be used to connect it to FPGA, also IO voltage. In future hardware revision I could make some changes for main board, that will help NV2 owners to bring alive this sensor too.
 

Offline Hydron

  • Super Contributor
  • ***
  • Posts: 1099
  • Country: gb
I agree that a visual camera in the MSX style (i.e. for edge enhancement etc) isn't necessary with a high resolution (>240p) thermal imager. If you're using the camera for security or something however it can be useful as a separate imager, e.g. to zoom in on someone that you've detected with a wider angle thermal imager.

I'm very impressed with this project - I know a little (mostly theoretical) about thermal imager design and it's a significant undertaking, normally involving a commercial team and significant money! Very interested to learn more.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I'm very impressed with this project - I know a little (mostly theoretical) about thermal imager design and it's a significant undertaking, normally involving a commercial team and significant money! Very interested to learn more.
Thanks!

I agree that a visual camera in the MSX style (i.e. for edge enhancement etc) isn't necessary with a high resolution (>240p) thermal imager. If you're using the camera for security or something however it can be useful as a separate imager, e.g. to zoom in on someone that you've detected with a wider angle thermal imager.
Integration of visual camera is quite challenging, though not impossible. The problem is that right now there is no free IOs on headers that connect boards with each other, while IO muxing is also not easy. I also doubt that, for example, in UAV mode this visual camera will be better that specialized FPV one. Though at the same time you could easily multiplex outputs of FPV visual and thermal camera. Generally, I think there should be really strong arguments for visual camera integration.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Main board (M-board):

1025286-0 1025302-1

The brain of the whole device is a Spartan-6 FPGA. As a memory buffer I decided to use HyperRAM. HyperRam memory controller was also developed from the scratch, as there was no high performance implementation of this IP-core in the opensource space. In fact this IP-core is my separate project.
You could ask me, why didn't I use free FPGA's integrated memory controller with DDR2/DDR3 memory. The thing is a very high power consumption, DDR3 + integrated controller would burn even much power, than the whole camera consums itself.

The red bond wire is a bug fix. Though is not the last bug. Using Spartan-6 was a mistake. This FPGA is quite powerful, but also old, has a very poor support, no updates, some bugs in IDE, endless loops in peripherial drivers... :palm:
That's why I'm planning to replace it with Spartan-7 XC7S50 in the same package. I hope the migration will be easy, as previously I had some experience with Xilinx 7-series FPGAs.

The whole stack:
1025290-2 1025294-3
« Last Edit: July 17, 2020, 01:49:39 pm by VGN »
 

Offline Hydron

  • Super Contributor
  • ***
  • Posts: 1099
  • Country: gb
The brain of the whole device is a Spartan-6 FPGA. As a memory buffer I decided to use HyperRAM. HyperRam memory controller was also developed from the scratch, as there was no high performance implementation of this IP-core in the opensource space. In fact this IP-core is my separate project.
You could ask me, why didn't I use free FPGA's integrated memory controller with DDR2/DDR3 memory. The thing is a very high power consumption, DDR3 + integrated controller would burn even much power, than the whole camera consums itself.
Commercial camera cores use things like PSRAM (psuedo-static RAM) for this reason also - lower power consumption than DDR2/3 plus controller.
 

Offline conmega

  • Contributor
  • Posts: 23
  • Country: us
The red bond wire is a bug fix. Though is not the last bug. Using Spartan-6 was a mistake. This FPGA is quite powerful, but also old, has a very poor support, no updates, some bugs in IDE, endless loops in peripherial drivers... :palm:
That's why I'm planning to replace it with Spartan-7 XC7S50 in the same package. I hope the migration will be easy, as previously I had some experience with Xilinx 7-series FPGAs.

It should be pretty simple yes... Stuff like clocking/serdes units have minor changes and such but I have done a bit of similar porting in the past. Even from Altera parts to Xilinx parts from tmbinc's Altera based FPDLink receiver over to a Xilinx 7 series part. It was only a few days of banging my head against the wall. Mostly because I had to learn migen :) Otherwise just some primitives that should need to be changed and some logic wrapped around possibly.
 

Offline ArsenioDev

  • Frequent Contributor
  • **
  • Posts: 275
  • Country: us
    • DiscountMissiles: my portfolio and landing page
Holy crap this is basically the HOLY GRAIL for my projects. I'll be watching this with INTENSE interest, it's essentially my dreams come true.
 

Offline Miek

  • Regular Contributor
  • *
  • Posts: 92
  • Country: gb
Since you're planning an FPGA swap anyway, it's worth considering the Lattice ECP5 series - the open-source toolchain for them is very good these days. Also, it would probably tempt more people to get involved once you open source the project.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Holy crap this is basically the HOLY GRAIL for my projects. I'll be watching this with INTENSE interest, it's essentially my dreams come true.
Thanks! All the fun is ahead  ;)

Since you're planning an FPGA swap anyway, it's worth considering the Lattice ECP5 series - the open-source toolchain for them is very good these days.
I was thinking about that long ago, when realized that swap is needed. Well, yes, open-source toolchain is very cool and important (especially for security based projects), but I'm not completely sure, that it is worth doing that. I have several arguments:

1. I've already started migration to S7. I'm also afraid that migration from Xilinx to Lattice will be not easy, nor fast. Whole design is AXI4/AXI4-Stream based. I use Xilinx DMA, SERDES... Generally, Lattice swap will cause too massive rework.

2. Lattice ECP5 40nm is a pretty good power optimized solution, but power consumption is not a problem even with Spartan-6 45nm. I hope Spartan-7 28nm will burn even fewer amount of power.

3. I have almost no experience with Lattice, so I'm a bit confused with ECP5 IO performance. ECP5 has 4 SERDES channels, while Xilinx has SERDES on each IO. Will it be enough for HyperRAM and HDMI simultaneous work? BTW, not also sure that ECP5 supports TMDS standard (for HDMI). Is it possible to stream HDMI directly from FPGA without a PHY? If yes, why do they use here HDMI PHYs with ECP5 for such a simple project?
http://www.latticesemi.com/-/media/LatticeSemi/Documents/UserManuals/1D2/FPGA-UG-02036-A.ashx?document_id=52240

Meanwhile Spartan-7 with 1080p 60Hz and even 1440p 30Hz output:
http://labs.domipheus.com/blog/hdmi-over-pmod-using-the-arty-spartan-7-fpga-board/

4. If we try to compare capacity somehow: LFE5UM-45 < XC7S50 < LFE5UM-85
LFE5UM-45 should be excluded, as it looks like to be equal to XC6SLX45, that I use now, but keep in mind that ECP5 is LUT4, while S6 and S7 are LUT6.
Obviosly, we should choose between XC7S50 and LFE5UM-85.

Digikey prices:
LFE5UM-85F-7BG381C ~38$
XC7S50-1CSGA324C    ~53$

Lattice price is lower, but not as much, also FPGA's price is not a big part of the total device cost.

Сorrect me if I'm wrong.


Also, it would probably tempt more people to get involved once you open source the project.
It will be open source anyway, please, be patient  :)
Let's resolve hardware issues first. I hope next week I will commit preliminary schematics (rev.C) for community review and discussion.
« Last Edit: July 21, 2020, 01:23:49 am by VGN »
 

Offline firehopper

  • Frequent Contributor
  • **
  • Posts: 408
  • Country: us
also would be nice for those uninitiated, what to look for to get these modules. the nv3 ones.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
also would be nice for those uninitiated, what to look for to get these modules. the nv3 ones.
66549322653 or 9322653 - BMW
A2229052805 - Mercedes
4G0980552/4G0980552A - AUDI/WV?
 
The following users thanked this post: Fraser, BravoV, Hydron, ArsenioDev, ajw107, figment

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
The focus system:

1028602-0

There are four SLA printed parts:

1028562-11028574-21028578-31028582-41028586-5

1. wheel
2. worm
3. motor mount
4. worm mount

Focus board:

1028566-61028570-7

One of the most tricky things was the connection of focus wheel with the lens holding ring. You cannot imagine how much I hate glue, that's why you didn't see any glue connection in the whole design. But...unfortunatelly there were no other cheap and easy way to connect this two parts. Small drops of cyanoacrylate on each tip of the focus wheel allows to reliably connect this two parts so, that I cannot break it by hand. I also tried epoxy, but the adhesion was very poor, so cyanoacrylate is the best for gluing this type of plastic. At the same time the whole focus mechanism is quite lightweight and there is no need to apply much force to make the lens move.

1028594-81028598-91028590-10

The lens holding ring thread allow to make 5 turns of the focus wheel, until it fully unscrews and falls out, that's why we can use 4 turns to adjust the focus. The focus distance changes non-linearly with the wheel position, and that is very good, because a single turn from the fully closed position allows to cover focus distances from ~30cm to infinity, while for macro you will need to make few other turns.

You can also see a photomicrosensor that allows to precisely control the lens position. It is mostly needed to keep the lens away from endpoints, preventing lens lock and unscrew. Also this sensor will be helpful for autofocus! The wheel has 25 teeth, the worm has a single thread and 4 blades, that cross the photomicrosensor slot, so if we detect both rising and falling edges of the pulse, we will have 25x4x2 = 200 points of lens position per a single turn, i.e. accuracy of 1,8 degree.

The focus board thickness is 3mm, it also has 2 additional support points (two 10mm white threaded spacers), that allows to keep sufficient construction rigidity.
The worm mount has a special brass sleeve, that helps to minimize the wear.

The focus motor is a 6mm dia coreless motor with 3 stage built in planetary gearbox (1:136). It is very powerful and consums few energy, but motor vibration is high and rigid connection to the PCB causes some sound amplification. But I have a solution for this, the motor mount is to be redesigned, I'm planning to add a special silicone cover for the motor that will damp the vibrations.

Two words about focus gears reliability. In fact I do not see any wear of the worm or wheel, because after proper UV curing this type of plastic becomes very strong. But I plan to make a continues stress test to define the possible wear level. Making this complex parts on SLA printer is quite cheap and at the same time  quality and reliability is high, that's why I'm planning to use this solution in release design. I have a small experienсe with SLA printing, that's why feel free to point out flaws.

Some common view:

1028606-111028610-121028614-131028618-141028622-151028626-161028630-17

Of course, all plastic parts STEP models will be avaliable.



P.S. no video yet, but maybe you can enjoy animation :)

« Last Edit: July 22, 2020, 03:46:50 pm by VGN »
 
The following users thanked this post: Fraser, encore2097, lukier, Syncronisator

Offline Ismsanmar

  • Regular Contributor
  • *
  • Posts: 53
  • Country: es
Perhaps the vibrations come from the lack of lubrication of the gears, apart from the slightly rough surface finish. You should apply some type of non-hygroscopic based grease, such as PTFE grease.
 

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
This is the most impressive thermal camera project that I have seen to date. You have manufactured a very professional mechanical and electronic solution. If I was working on your PCB’s I would honestly think that they were the product of one of the well known thermal camera manufacturers  :-+ Most impressive  and I can only dream of having such design skills  :-+ :-+

Fraser
« Last Edit: July 22, 2020, 09:22:10 pm by Fraser »
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 
The following users thanked this post: VGN

Offline Vipitis

  • Frequent Contributor
  • **
  • Posts: 880
  • Country: de
  • aspiring thermal photography enthusiast
That is so beautifully engineered, my plans for a 3D printed follow focus (external) that I sketched out during some downtime at an exam look pathetic.

I will take a deep look at this completely project and see which parts I can somehow use for my project. Experience and tools is what I feel I am lacking.
 
The following users thanked this post: VGN

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Perhaps the vibrations come from the lack of lubrication of the gears, apart from the slightly rough surface finish. You should apply some type of non-hygroscopic based grease, such as PTFE grease.
When I disassembled this motor, I found out that there was no any grease at all in the gearbox. I tried to apply a bike grease (the only grease I had), but things got worse, the "no load" current increased. Looks like too much grease in the gearbox prevented normal rotation, amount of lubrication must be proper. On the other hand, by default at max voltage (3.0V) the shaft rotates at 240RPM, so with ratio 1:136 we have motor rotating at about 32640 RPM, this is a lot. I suppose that probably vibration is caused by rotor disbalance.

This is the most impressive thermal camera project that I have seen to date. You have manufactured a very professional mechanical and electronic solution. If I was working on your PCB’s I would honestly think that they were the product of one of the well known thermal camera manufacturers  :-+ Most impressive  and I can only dream of having such design skills  :-+ :-+
Many thanks for your support! :)

That is so beautifully engineered, my plans for a 3D printed follow focus (external) that I sketched out during some downtime at an exam look pathetic.
Thank you! Just keep up. If your design or piece of a code starts looking pathetic for you, that means your skills are growing, this is a good sign. :-+ Wish you luck with your project! ;)

I will take a deep look at this completely project and see which parts I can somehow use for my project. Experience and tools is what I feel I am lacking.
Feel free to use any piece of this design. I'm going to start commiting some sources by the next week. I'll leave a link to my github repo.
 
The following users thanked this post: zrq

Offline dnhkng

  • Contributor
  • Posts: 16
  • Country: de
Soooo, VGN is the initials of three people, right? Vince, Gary, and Nancy? Because this project seems to be the work of a team of hardware designers, FPGA engineers, and electronics engineers all working together  :-+

Really impressive work, bravo!
 

Offline GeorgeOfTheJungle

  • Super Contributor
  • ***
  • !
  • Posts: 2699
  • Country: tr
The further a society drifts from truth, the more it will hate those who speak it.
 

Offline Flanbix

  • Contributor
  • Posts: 30
  • Country: gb
  • If you don't know, ask. If you know, share.
This project is great, lots of features. It is looking quite professional !  :-+
Keep the image and details comming.
 

Offline ajw107

  • Newbie
  • Posts: 6
  • Country: gb
Just wanted to say I joined this forum just to say thank you, and how impressed I am with your design.  I can;t wait until we can order the components.  Just got myself a new Audi 4GO980552A camera off eBay for £295 so I can do this project.  Should be buy the FPGA now as well, or will you be selling all the circuit boards as a kit?
If any one is interested in getting the same camera, there is one left at that price (less than a half the normal price for a used one) and the link is:
https://www.ebay.co.uk/itm/293490373204
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Soooo, VGN is the initials of three people, right? Vince, Gary, and Nancy? Because this project seems to be the work of a team of hardware designers, FPGA engineers, and electronics engineers all working together  :-+
Really impressive work, bravo!
Thank you! Hahaha, not Gary, me and Nancy call him Gabe ;)

This project is great, lots of features. It is looking quite professional !  :-+
Keep the image and details comming.
Thanks!

Just wanted to say I joined this forum just to say thank you, and how impressed I am with your design.  I can;t wait until we can order the components.  Just got myself a new Audi 4GO980552A camera off eBay for £295 so I can do this project.  Should be buy the FPGA now as well, or will you be selling all the circuit boards as a kit?
If any one is interested in getting the same camera, there is one left at that price (less than a half the normal price for a used one) and the link is:
https://www.ebay.co.uk/itm/293490373204
Thank you! I'm really very glad, that you and other people are interested in this project, that additionally motivates to continue. Yes, I have plans to make a some kind of kit, that anyone can buy. This kit will include everything you need, except the thermal camera, sd card, battery (due to some problems with air transportation of batteries). Later I will determine the exact kit contents. There is no need to buy a FPGA for this project. The design of this camera is developed in a way that you will not need even a solder iron to put all parts together. I will also make efforts to keep the functionality and price competetive with other brands, though that will be not easy.
« Last Edit: July 26, 2020, 09:14:57 am by VGN »
 

Offline zrq

  • Frequent Contributor
  • **
  • Posts: 343
  • Country: 00
This project is one of the most exciting projects I seen on this forum, I never expected a personal project can be so complete and elegant.
I'm considering buying a second hand BMW camera for the FPA, I wonder is there any pitfalls to avoid?
I would be also quite interested in the kit (given the price is reasonable  ;) ), as I don't have the equipment and experience to work on BGA packages.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
This project is one of the most exciting projects I seen on this forum, I never expected a personal project can be so complete and elegant.
I'm considering buying a second hand BMW camera for the FPA, I wonder is there any pitfalls to avoid?
I would be also quite interested in the kit (given the price is reasonable  ;) ), as I don't have the equipment and experience to work on BGA packages.
Thank you!  :)
Concerning pitfalls, I have described some of them in my previous post:
https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/msg3134494/#msg3134494
Actually, if the front window is not cracked and no one disassembled the camera, it should be in a good condition. If the window is cracked, the probability of sensor and lens survival is still quite high, but of course not 100%. As you can see two of my cameras with cracked windows have a good condition of lens and the sensor is working too. It's mostly a matter of price and luck. You can buy a new one for 300-500$ or take a chance with <100$ camera with cracked window. I bought two with cracked windows only because the price was low, I didn't expect that they will be working.

Concerning the kit, I believe you will not need to have any extra tools or skills to build and even calibrate the camera yourself.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Here some more images, captured right by camera. Original images were in .bmp format, as this is the easiest way for image capturing, but I had to convert it to .png, as eevblog forum do not support .bmp attaches. Of course in future I plan to add png support right on camera, as bmps are much larger.

P.S. videos are comming soon!
« Last Edit: July 27, 2020, 07:26:11 pm by VGN »
 

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
You cannot beat a decent microbolometer coupled with a good lens  :-+

At least you get a decent, low noise, image to process :) Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

Fraser
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 

Offline ajw107

  • Newbie
  • Posts: 6
  • Country: gb
Thank you! I'm really very glad, that you and other people are interested in this project, that additionally motivates to continue. Yes, I have plans to make a some kind of kit, that anyone can buy. This kit will include everything you need, except the thermal camera, sd card, battery (due to some problems with air transportation of batteries). Later I will determine the exact kit contents. There is no need to buy a FPGA for this project. The design of this camera is developed in a way that you will not need even a solder iron to put all parts together. I will also make efforts to keep the functionality and price competetive with other brands, though that will be not easy.

That is great news, as I'm no good with a soldering iron (rubbish eyesight), although i will keep on trying to give it a go, much to the deprement of my fingers.  I have to admit, I have also just gone and bought a new SLA printer (Kelant S500, bit of a gamble as no reviews - but on paper is sounds good for the price), as my old ANET8 has long since given up the ghost and I've been itching to try an SLA for years.  The photos are great by the way, is that a brain cross-section in photos 11-14? I know it can't be, but it looks like it, he he!  Looking forward to being able to order the curcuitrs when they become available, so excited, like a kid with a new toy.
 

Offline ajw107

  • Newbie
  • Posts: 6
  • Country: gb
Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).
This thread is actually the guide. :D
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...



You cannot beat a decent microbolometer coupled with a good lens  :-+
At least you get a decent, low noise, image to process :) Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

The lens definitly matters. I don't have Flir E8 right now, but I remembered that earlier I have made a some photos at home. Here is the same bathroom with OpenIRV and FLIR E8 (hacked E4). Of course, shooting time and some other conditions are different:
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...



Earlier I showed some images, shooted in good conditions, when the scene tempretature difference is high. As you could see, there is no noticeable noise.
Now, let's look at abilities of this camera in poor conditions.

First, so that you can better understand what's going on, here is a photo of bicycle at the wall with a cup of warm water. The room temperature is +25.5 °С , the water in cup is +29.5 °С (measured by external thermometer). Temperature difference of the coldest and hottest objects is about 4-5°С. As you can see, the image is quite good, further we will analyze this scene without a cup of warm water.





The same scene without a cup of warm water. This gif is a sequence of images shooted with a small time gap in a raw mode (no any kind of filtering):



As you can see, the noise gets out, as temperature range is probably about ~1°С. The noise pattern is complex and quite specific. After reading tons of literature and some analysis and I came to think that there are two main noise components:
1. Row noise. This is a temporal noise, i.e. random fluctuations of the whole pixel line, that randomly changes every moment.
2. Column noise. This is a FPN (fixed pattern noise) that changes quite slowly and cannot be fully removed even by FFC (flat field correction) procedure.



Earlier I have developed a special averaging module that helps to average a sequence of multiple frames comming out of the sensor. Max possible number of averaged frames is 262144, though reasonable values are 2, 4, 8, 16, 32, 64, 128. Signal averaging is a common technique, intended to increase the strength of a signal relative to noise. So, let's try to use this module with different averaging values:



We can see that starting from x8 value the row noise disapears, that confirms the nature of row noise. At the same time column noise is still present, averaging do not help to neutralize this type of noise, we need something more.



Few days ago I found a way to remove this column FPN in frequency domain by FFT (fast fourier transform) and a custom filter. As a test scene I decided to use some uniform object. Here you can see how it works. Image from camera (uniform body) is on the left, FFT of this image is on the right. Black lines on the FFT plot are the actual filter function, that "cuts out" the noise components. Here is the link to the online FFT tool that I used for experiments: https://www.ejectamenta.com/Fourifier-fullscreen/





At last I decided to test both averaging and FFT filters at the complex scene with a lot of details:



Also keep in mind that this experiments are done on equalized 8-bit image. I think that final image quality will be a little better if I apply FFT filter to the raw 14-bit pixel frames.



I don't want to start developing this FFT filter right now, it will be easier to develop it for new hardware and new FPGA. Though averaging is a good filter, it reduces the frame rate, that's why I should implement something else for row noise, of maybe FFT will be enogh, the time will show. At least I know how to deal with this noise. If you have good ideas and suggestions for image filtering, please don't keep in secret ;)



P.S. This is how lepton 80x60 "sees" this bicycle:  ;D

« Last Edit: July 30, 2020, 11:27:18 am by VGN »
 

Offline zrq

  • Frequent Contributor
  • **
  • Posts: 343
  • Country: 00
For the similar noise I noticed on Xtherm cameras, I wrote a routine to reduce the single column noise by filtering in the temporal space.

Code: [Select]
merit = np.abs(cv2.Laplacian(cv2.GaussianBlur(imgFlt, (3, 3), 3), cv2.CV_32F)) / 65536
merit = 1 / pow(merit + 0.005, 3)
leftErr = imgFlt[:, 0:leftHalf] - cv2.GaussianBlur(imgFlt[:, 0:leftHalf], (1, 5), 10)
horiLeftAcc = np.average(leftErr, weights=merit[:, 0:leftHalf], axis=1)
rightErr = imgFlt[:, leftHalf:] - cv2.GaussianBlur(imgFlt[:, leftHalf:], (1, 5), 10)
horiRightAcc = np.average(rightErr, weights=merit[:, leftHalf:], axis=1)
vertErr = imgFlt - cv2.GaussianBlur(img, (5, 1), 10)
vertAcc = np.average(vertErr, weights=merit, axis=0)
for n in range(0, np.size(img, 0)):
    imgFlt[n, 0:leftHalf] = cv2.subtract(imgFlt[n, 0:leftHalf], (k[0] * horiLeftAcc[n])).squeeze(1)
    imgFlt[n, leftHalf:] = cv2.subtract(imgFlt[n, leftHalf:], (k[1] * horiRightAcc[n])).squeeze(1)
for n in range(0, np.size(img, 1)):
    imgFlt[:, n] = cv2.subtract(imgFlt[:, n], (k[2] * vertAcc[n])).squeeze(1)

However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked. :-\
 
The following users thanked this post: nikitasius

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked. :-\
I made the same mistake at first)
I also think that parametrizing this filter pattern like this will help to find optimal configuration:





By the way, I found a cool Protoshop plug in for FFT noise reduction.
http://ft.rognemedia.no/

I tested it on bicycle image, drawing that cross filter pattern by hand. Now no image crop, no averaging, only pure FFT filter.

Before:



After:



You can see some luminance fluctuation in filtered image, I'm sure that this is caused by changes in histogram equalization, caused by noise peaks. That's why we should apply this filter on 14-bit raw image before histogram gain correction. Anyway, there is almost no high frequency row/column noise and that is really good!
« Last Edit: July 30, 2020, 11:03:38 am by VGN »
 

Offline ajw107

  • Newbie
  • Posts: 6
  • Country: gb
This thread is actually the guide. :D
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...
You know, I thought that as soon as I pressed the post button (that's what I get for asking questions when I'm half asleep).  In my head there was some magical M12 CAN -> USB connector, he he!  Any news on when things will be ready for us to try (no rush, just over excitment, like a child with a new toy)!
 

Offline Hydron

  • Super Contributor
  • ***
  • Posts: 1099
  • Country: gb
These noise issues are present on other microbolometers too - e.g. the ULIS 17u 640x480 FPA has a well known issue with column noise, and there are some filtering techniques that have been tried to improve it (some suggested by ULIS themselves I think). I'm not sure whether I have any info about these, but if I do and they are able to be distributed (unfortunately unlikely) I'll do so.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hello!

Sorry for the delay, I was working at the next hardware revision. I have a lot of progress and updated. But first, the video capture, that I have promised long ago:



I doubt that youtube will ever show this video in original quality. So, for those of you, who would like to see the original quality and frame rate, here is the link to my google drive (video file is about 180MB): https://drive.google.com/file/d/1QfK6TBKxTnJjb9-Vjo1R4MsX56F_mhFq/view?usp=sharing
Google drive will suggest you to watch it online in browser, but you should better download it.

Video was captured over USB. Note that this thermal camera do not record the sound. The sound was added later.
« Last Edit: August 19, 2020, 12:55:58 am by VGN »
 
The following users thanked this post: Jay_Diddy_B, Syncronisator

Offline Jenny

  • Contributor
  • Posts: 25
  • Country: us
Hi, I’m so amazed (like others also do) at how incredible work you have done.  :-+
Just a quick question regarding to the last video: I realized you have motorized focus on the camera? Is it auto or manual? I found it is a bit slow, but stop at the focal point without overshoot. Is it phase detection or even laser assisted?
Again, it’s really unbelievable how you done such work, in a short time...
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hi, I’m so amazed (like others also do) at how incredible work you have done.  :-+
Again, it’s really unbelievable how you done such work, in a short time...

Thank you! I'd not say, I have done it in a short time.. ::)

Just a quick question regarding to the last video: I realized you have motorized focus on the camera? Is it auto or manual? I found it is a bit slow, but stop at the focal point without overshoot. Is it phase detection or even laser assisted?
Yes, the focus is motorized, you can find focus system PCB photos in my previous posts. There are two modes, manual and auto. On the video, you saw a manual mode, I changed the focus myself. Autofocus fuction is under development now. It is based on image sharpness detection. Right now it is implemented in software and works too slow and not precize. I definitly should make a hardware acceleration of image sharpness calculation. In this case I will be able to detect sharpness gradient change at the highest frame rate, and accidental sharpness change due to the hand shake will not confuse the algorithm, I hope.

Regarding to the speed, yes it is slow now, because I have limited the RPM. The focus motor is rated for 3.0V, but it is running at ~2.0V now. Both shutter and focus motors are powered almost directly from Li-ion battery, which voltage is variable (4.2V-3.0V) and a little higher than motor recommended voltage. As I don't want to overvoltage the motor power, I'm using PWM to limit the voltage. Further I'm going to adjust the PWM duty cycle according to the battery voltage to keep it at 3.0V.

There is also another feature. We have 4 lens turns to adjust the focus. The focus distance changes non-linearly with the lens position, and that is very good, because a single turn from the fully closed position allows to cover focus distances from ~30cm to infinity. Lens movement under single turn will be pretty quick, thought for macro mode you will have to make few other turns.

Forgot to say, there is one more way to increase the focusing speed by using motor with smaller gear ration. Right now reductor ration is 1:136. But there same type of motors with 1:26 gearbox, with higher RPM. If torque will be enough, it will be worth replacing the motor.
« Last Edit: August 19, 2020, 03:46:17 pm by VGN »
 
The following users thanked this post: Jenny

Offline ajw107

  • Newbie
  • Posts: 6
  • Country: gb
I have to admit, I was getting a little worried when we hadn't heard from you in a while, so pleased it is going well.  The video quality is incredible, it seems to have come on leaps and bounds from the still very good bike photos we had earlier.  Wonder if I could ask a few questions:
1)  Will the FPGA be able to handle the thermal data processing and the auto focus and keep the frame rate?  I'm not familiar with the Spartan-6 board?
2) Is there an ETA on when we can get a look at some of the code and stl files?  I'm just being impatient with excitement, I know.  But The code will be fascinating to look through for the image processing alone, and the stl files will be a good test for my printer (whenever it eventually arrives, grrr).
3) I'm not entirely sure where abouts you are in your project roadmap, it seems you are getting close to a finished product, just requiring some some tweaks (sometimes the phase that takes the longest time, but hopefully everyone on here can help once it goes on github).
4) Any idea of how much the kit minus the nv3 will cost?  I'm moving house at the moment, so counting the pennies, and want to make sure I put some money aside.  Even better if you could say some thing like the circuitry will cost about X and the housing/gears/internal parts should be about Y, etc.

Anyway, as I say, magnificent work, it really is impressive what you have accomplished.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I have to admit, I was getting a little worried when we hadn't heard from you in a while, so pleased it is going well.
Sorry for this, I will try not to make long delays or at least warn about that.

1)  Will the FPGA be able to handle the thermal data processing and the auto focus and keep the frame rate?  I'm not familiar with the Spartan-6 board?
New hardware will be based on newest Spartan-7. Of course, FPGA will be able to handle the thermal data processing, autofocus and many other things simultaneously. FPGA is very good with paraller processing.

2) Is there an ETA on when we can get a look at some of the code and stl files?  I'm just being impatient with excitement, I know.  But The code will be fascinating to look through for the image processing alone, and the stl files will be a good test for my printer (whenever it eventually arrives, grrr).
3) I'm not entirely sure where abouts you are in your project roadmap, it seems you are getting close to a finished product, just requiring some some tweaks (sometimes the phase that takes the longest time, but hopefully everyone on here can help once it goes on github).
Don't get me wrong, this is my first open source project. Though I have developed quite a lot of different devices in a small teams with private repositories, I don't have an experience with open source and wide community. I'm not sure I know the proper way of organizing such kind of projects. Is it better to make a public repo from the very beginning or keep it private for those of you, who would like to collaborate and reveal sources with the first batch of devices? I've always thought about it. Maybe you or anybody else can give an advice?

Also I didn't expect that I will make so many changed for the new revision, this is another reason that I haven't published schematics yet... :-\. Tomorrow I will tell about most of new features.

Regarding to the technical side of the project, in general terms, this is my plan for nearest future:
1. Finish new hardware revision schematics (will be finished in a week).
2. Purchase components and new PCBs (shipping is about 2-3 weeks).
3. Rewrite the HyperRAM memory controller for Spartan-7 and publish it as a separate project.
4. By this time I hope to get new PCBs and component and can assembly everything.
5. Start porting the design to new hardware.

If everything is ok at this step and I'm sure that hardware is working properly, I could make a first small batch of devices for developers and testers. After that I think we will be ready for larger production. This is a matter of discussion.

4) Any idea of how much the kit minus the nv3 will cost?  I'm moving house at the moment, so counting the pennies, and want to make sure I put some money aside.  Even better if you could say some thing like the circuitry will cost about X and the housing/gears/internal parts should be about Y, etc.
I will provide this information as soon as I get the final BOM. But I'm going to keep the price competitive to HT-301, TE-Q1, HT A2, FLIR C2 and other "cheap" thermals, though OpenIRV is going to be more powerful and multifuctional. Also keep in mind that the batch size makes difference on the final price.
 
The following users thanked this post: zrq

Offline railrun

  • Regular Contributor
  • *
  • Posts: 115
If you are looking for tester I would be happy to be a tester.
 
The following users thanked this post: VGN

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
@VGN
Looks very interesting.

Do you know if a 4H0980552 is an acceptable version.  Note the H instead of G.  How would I find out what that means?

thanks
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
If you are looking for tester I would be happy to be a tester.
I'm very glad, thank you! That will be cool if you and all other people, who would like to be a tester or even a developer of the first kit batch, email me (view my profile for email).

Do you know if a 4H0980552 is an acceptable version.  Note the H instead of G.  How would I find out what that means?
I don't know what actually each letter means, but according to the photos, 4H0980552 is the previos generation (NV2). This project initially was designed for NV3, but I have plans to make hardware in a way to support more types of sensors.
 

Offline tmbinc

  • Frequent Contributor
  • **
  • Posts: 253
The easiest take-away is the connector - it's always this HSD ("weird automotive never seen before") connector, and either there are two additional pins next to it (NV2) or not (NV3). NV2 has Power (separate), CAN and LVDS so it needs 6 pins, whereas NV3 has Power and this "weird automotive never seen before bidirectional highspeed interface that's neither CAN nor LVDS but can replace both", and hence only needs 4 pins.

The case is also distinctively different, but harder to describe/see.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
The easiest take-away is the connector - it's always this HSD ("weird automotive never seen before") connector, and either there are two additional pins next to it (NV2) or not (NV3). NV2 has Power (separate), CAN and LVDS so it needs 6 pins, whereas NV3 has Power and this "weird automotive never seen before bidirectional highspeed interface that's neither CAN nor LVDS but can replace both", and hence only needs 4 pins.
The case is also distinctively different, but harder to describe/see.
Exactly!
 

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
Thanks for the help.  Had a seller on eBay tell me they were the same.  He said the 4H was from an Audi A8 instead of A7.  Luckily I saw your quick responses and hadn't paid yet. 

thanks.
 

Offline tmbinc

  • Frequent Contributor
  • **
  • Posts: 253
Well for NV2 we have https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/, but VGN's work of course is far more impressive.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Well for NV2 we have https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/, but VGN's work of course is far more impressive.
tmbinc, btw I have to notice that I saw your investigation of NV2 on debugmo.de long ago. That is an incredible work that inspired me to work on NV3! I'm a newbie on this forum, I haven't remembered your nick-name, so I only recently realized that was you, haha))  ;D
Long story short, very glad to meet you!
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I'm finishing new hardware revision and want to tell you about main changes:

1. Most of the changes are refered to M-board (main). I have replaced old Spartan-6 XC6SLX45 with new Spartan-7 XC7S50.

2. I decided to add more RAM. I found new HyperRAM part which is twice cheaper and +20% faster than the previous one. The reason of adding more RAM is that probably image filtering with FFT will require more memory bandwidth and I'm afraid that single channel RAM will not be enough. Anyway it is always better to have a footprint for an additional memory rank, than do not have it at all.

3. From some point I realized that it is worth to support or, better say, give a chance to support more types of sensors than only NV3's one.

NV3's sensor ISC0901B0 is connected to the main board over 40-pin connector. There are:
  • 1 pin is connected to sensor frame (chassis)
  • 2 pins for bias power (11.0v)
  • 2 pins for core power (3.3v)
  • 2 pins for io power (2.5v)
  • 6-pins to control the sensor. clk, ena, cmd, bias, data_odd, data_even. Details further.
  • 2 pins are NC (not connected to sensor die and anywhere else)
  • 25 pins (remaining) are ground
Well, leaving 25 pins for ground is ridiculous, that's why I have developed a new backward compatible pinout that will use some ground pins as IOs and connected them to FPGA. If this extended IOs will be configured as input at FPGA side, the direct path to ground over ISC0901B0 board will not stress FPGA's bank.

Generally now we have 18 single ended IOs that are connected to a single FPGA bank. The bank power can be configured in safe range (0.6v - 3.3v) over DC-DC's feedback resistor divider. Other board curcuits have separate power, that's why IO voltage change will not affect anything else. I hope these amount of IOs will be enough to connect any sensor. Probably it is worth to route some of this IOs in differential line manner, but I'm not sure. Any ideas or suggestions are welcome.

Also two NC pins I have connected to system power (battery rail) as an additional power path.
Bias and core power rails are also configurable or can be disabled if desired.

4. I have reworked whole power system, removed 2 LDOs and improved efficiency of some power rails.

5. You can also see some strange 10-pin blue connector. There are 7 IOs (3.3V fixed), ground and two power rails (regulated 3.3V + battery rail). It can be used for any general purpose, but I have one quite interesting task for this connector, details will be furter.



Regarding to the support of different sensors. We have a base part and any other sensor can be connected over special adapter board. Optionally we will also need a cover to protect the sensor. The most common sensors that I think is possible to support, except NV3's are NV2 sensor (ISC0601B), probably Seek Thermal Mosaic Cores and of course Lepton2/3 (though this hardware for such a tiny core is an overhead...)
« Last Edit: August 22, 2020, 01:18:50 am by VGN »
 
The following users thanked this post: zrq

Offline tmbinc

  • Frequent Contributor
  • **
  • Posts: 253
VGN, thanks! I'm deeply impressed by your reverse-engineering, electrical design, industrial design _and_ project presentation skills. I can't wait to use your design and play around with it!

I think there are great benefits of this project that go far above "getting a cheap thermal camera" (which is a very great goal in itself!). It will bootstrap open source efforts to operate on raw thermal sensor data and I see a realistic chance of building an image processing pipeline that matches (and exceeds) that of commercial products. The math is not trivial, but also not super hard, and it is about time that we don't rely on companies that tend to intentionally cripple image quality (noise generator, mandatory fusion and cropping - WTF) on the affordable range of thermal imagers, and take this back into our hands.
 
The following users thanked this post: VGN

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Possible configuration without the focus board. In this case we should change the bottom enclosure part. It has the same outline, but different height (13mm to 7mm).
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Welcome to my GitHub repo!   ;)

https://github.com/OVGN/OpenIRV

All mechanical parts STEP models are avaliable. You can try to print some of them for a test. That will be cool if you show your attempts to us, especially gears. I have used SLA printing, but didn't try any other technologies. Keep in mind that this parts are not release version, I'm going to make some changes, even not backward compatible.
« Last Edit: August 25, 2020, 11:25:14 pm by VGN »
 
The following users thanked this post: Fraser, tmbinc, lukier, ajw107

Offline ajw107

  • Newbie
  • Posts: 6
  • Country: gb
That is wonderful news, it's quite late here now but just got my SLA today, so will definitely have a play when I get round to assembling it later this week.





Welcome to my GitHub repo!   ;)

https://github.com/OVGN/OpenIRV

All mechanical parts STEP models are avaliable. You can try to print some of them for a test. That will be cool if you show your attempts to us, especially gears. I have used SLA printing, but didn't try any other technologies. Keep in mind that this parts are not release version, I'm going to make some changes, even not backward compatible.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Thanks to Treehouseman, looks like even forum users could not email me, that's strange.

So, here is my email both for registered and non-registered users. I hope this "light" obfuscation will protect me from spam somehow  :D
Consider to email me if you would like to be a beta tester, developer or just a backer.
It would be great if you keep mail header like this: "OpenIRV.Developer/Tester" or "OpenIRV.Backer" or etc...

I promis not to publish your email anywhere else and do not send mails, that are not relevant to OpenIRV project.

 

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
I just received an e-bay NV3 that seems in near new condition with just a plastic tab on the window holder broken.  Opening the case reveals a ribbon cable from below going up to the window that is not shown in your pictures.  What is that for?  I can probably get a picture, but its pretty tight.  How do I unhook it?
« Last Edit: August 27, 2020, 01:10:43 am by bicycleguy »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I just received an e-bay NV3 that seems in near new condition with just a plastic tab on the window holder broken.
This window holder locking plastic tab breaks very easily. Mine was broken too)

Opening the case reveals a ribbon cable from below going up to the window that is not shown in your pictures.  What is that for?  I can probably get a picture, but its pretty tight.  How do I unhook it?
This is actually an internal heater with integrated SMD thermistor. Original camera activates this heater in cold weather conditions to melt the ice at the front window, as ice is not IR transparent. There is a FPC connector at the other end of this cable. The heater itself is glued to the plastic housing by double-sided tape. You don't need this part with OpenIRV.
« Last Edit: August 27, 2020, 02:07:09 am by VGN »
 

Offline Treehouseman

  • Supporter
  • ****
  • Posts: 58
I just received an e-bay NV3 that seems in near new condition with just a plastic tab on the window holder broken. 

So you're the one that bought that camera! I went to go grab it the other day because 5% ebay bucks and it was gone. Ended up going for the Mercedes one instead.
 

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
Here's an attempt to print the worm with a regular printer, Prusa Mk3s.  Looks bad but in my experience  it will probably still work.
" alt="" class="bbc_img" />
 

Offline ArsenioDev

  • Frequent Contributor
  • **
  • Posts: 275
  • Country: us
    • DiscountMissiles: my portfolio and landing page
Damn you lot snapping up all the good deals on them, back to monitoring the alerts smh
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Here's an attempt to print the worm with a regular printer, Prusa Mk3s.  Looks bad but in my experience  it will probably still work.
Nice try!) I counted 31 layer per 5.1mm or looks like your layer height is around 0,15mm. I think you probably can get much better quality with FDM if you make layer height 0,05mm, exact this layer height I used while printing my parts on SLA.
 

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
Nice try!) I counted 31 layer per 5.1mm or looks like your layer height is around 0,15mm. I think you probably can get much better quality with FDM if you make layer height 0,05mm, exact this layer height I used while printing my parts on SLA.
Yes on both.  Never tried smaller than 0,15.  I have a 0,3 nozzle to try too.
Here's an uncleaned-up gear for test.


Just curious, is it necessary to use black?
« Last Edit: August 29, 2020, 07:44:22 pm by bicycleguy »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Yes on both.  Never tried smaller than 0,15.  I have a 0,3 nozzle to try to.  Here's an uncleaned-up gear for test.
Yes, this one looks much better! But I'm not sure about that four tips. Anyway, FDM looks promissing.
Just curious, is it necessary to use black?
Do you mean plastic color? What's the difference?



BTW, I haven't still tell about the lens internals. If you unscrew the lens holding ring, you can find a lot of interesting parts. We are going to use only the lens and the lens holding ring (first two positions on photo). Also we need some spring to compensate the thread backlash. This is extreamely needed for proper autofocusing, as backlash causes an error when the motor changes rotation direction. The lens movement distance from end to end is 2mm, and we also have 2mm of space from the bottom of the housing. Generally we need a 4mm height light spring with 2mm working stroke. Unfortunatelly, we cannot reuse original wave spring, as it is too stiff, because it was designed only to lock the holding ring from being unscrewed.

Frankly speaking, that was a great headache, as manucturing a new special spring could be too costy. Manufacturers just hangs up the phone if you are going to purchase less than thousands of spring... :( But luckily I found a solution - stencils! :) The stencils, that are usually used to transfer solder paste to a bare PCBs. I decided to design a flat spring, based on this technology. This is going to be reliable and very cheap! Another good feature is that stencils are stainless.

I haven't finished the spring design yet, I should define the number, the width and the length of "legs", the stencil thickness. There are some early examples in attachments. Maybe you noticed a mistake on renders, that legs ends should be bended too.
« Last Edit: August 29, 2020, 10:27:07 pm by VGN »
 

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
Just curious, is it necessary to use black?
Do you mean plastic color? What's the difference?
The inside of the gear beveled surface is nearly a lens shade.  On a visible light camera this would all be black.  Is this necessary?  Almost seems like white would emit less IR.  But I'm ignorant of this IR stuff.

I have designed a few stainless steel springs however.  The material is critical.  Not sure about stencil material.  If you can bend it sharply and it doesn't spring back its the wrong kind.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
The inside of the gear beveled surface is nearly a lens shade.  On a visible light camera this would all be black.  Is this necessary?  Almost seems like white would emit less IR.  But I'm ignorant of this IR stuff.
The camera FOV is 24° h x 18° v, according to this documentation: http://www.safetyvision.com/sites/safetyvision.com/files/FLIR_PathFindIRII_User_Guide_1.pdf
The angle that this inside of the gear beveled surface forms with main optical axis is wider, than FOV angle of 24° h. In other words, camera do not see any presence of this gear, as it doesn't cross the FOV. That's why I think there is no sense to change the plastic color.

I have designed a few stainless steel springs however.  The material is critical.  Not sure about stencil material.  If you can bend it sharply and it doesn't spring back its the wrong kind.
I believe stencil material and thickness should be stable, as this parameters are very critical for PCB manufacturing. Anyway, this is a matter of experiments.
Looking for good ideas..
 

Offline Ruhkukah

  • Newbie
  • Posts: 1
  • Country: ru
Barev, VGN! Hi all!

I took a stab to print these two challenging focus parts (worm and wheel). Part of the worm broke off after my kid played with it (to speed things up I used 5% infill, so it is pretty fragile)  :-DD Now reprinting it
IMO the quality is good enough for the purpose, both parts are very smooth to the touch. For the worm I used black PLA filament, 0.05mm layer height, 0.15mm TriangleLab nozzle and 0.15mm extrusion width, 2 perimeters. For the wheel I tried 0.025mm layer height. My printer is Prusa MK3s stock. I didn't use any supports, so the cross and the circular plate needed a little bit of cleaning.
See attached.

VGN, I applaud your work - it looks very professional and top notch in every aspect. Looking forward to become a tester!

Pavel
 

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 55
  • Country: de
Can we use with this interface the Autoliv NV4 (12µm, 640x480, 50°x39°) sensor in the future ?
Would be cool when these sensor become available and we could use same interface.
 

Offline tmbinc

  • Frequent Contributor
  • **
  • Posts: 253
1. Is the FLIR PathFindIR II _identical_ with the Autoliv NV3? I know it looks pretty much spot on, but I wonder if it's the same thing. Is that an AutoLiv OEM product then? If so, is there also an NV2-based PathFind device?

I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.

2. NV4 - I've heard about it, but haven't seen it. Do you have any details? Is the NV business still at Autoliv or now at Veoneer?
 

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 55
  • Country: de
Veoneer is an Autoliv spin-off.
Unfortunately I don't have more information on the NV4 sensor.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Barev, VGN!

I took a stab to print these two challenging focus parts (worm and wheel). Part of the worm broke off after my kid played with it (to speed things up I used 5% infill, so it is pretty fragile)  :-DD Now reprinting it
IMO the quality is good enough for the purpose, both parts are very smooth to the touch. For the worm I used black PLA filament, 0.05mm layer height, 0.15mm TriangleLab nozzle and 0.15mm extrusion width, 2 perimeters. For the wheel I tried 0.025mm layer height. My printer is Prusa MK3s stock. I didn't use any supports, so the cross and the circular plate needed a little bit of cleaning.
See attached.
Barev, Ruhkukah!) Thank you for your test! Looks really good. You'd better try to use 0.025mm layer height for worm too. In this case the worm surface will not be so stepped. Looking forward your results)
VGN, I applaud your work - it looks very professional and top notch in every aspect. Looking forward to become a tester!
Thanks much! Don't forget to email me to become a tester. You can find my email in attached images of the first topic post.



1. Is the FLIR PathFindIR II _identical_ with the Autoliv NV3? I know it looks pretty much spot on, but I wonder if it's the same thing. Is that an AutoLiv OEM product then? If so, is there also an NV2-based PathFind device?
I don't exactly know, but I think, that NV3 is identical to FLIR PathFindIR II with some light changes, that each car manufacturer make to fit this camera in their car.



Can we use with this interface the Autoliv NV4 (12µm, 640x480, 50°x39°) sensor in the future ?
Would be cool when these sensor become available and we could use same interface.
I'm trying to do the best to provide as wide support of different IR cores as it is possible: more RAM, higher RAM throughput, more IOs at the sensor connector, more various adjustable power rails at the same connector, etc. In case of best scenario, we will just have to make a cheep adapter board to connect current hardware with a new sensor and develop a HDL module that will interface with it. Also, probably, a new sensor covering enclosure.
 

Offline horstmannhid

  • Contributor
  • Posts: 26
  • Country: de
Is it maybe an idea to use a ring magnet instead of the spring for the lens thread compensation? I think for 2mm there is always enough force available.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Is it maybe an idea to use a ring magnet instead of the spring for the lens thread compensation? I think for 2mm there is always enough force available.
You are right, this is definitly a good idea!  :-+

Four 1.5mm height magnets generate enough force, pulling a metal ring, mounted on the lens. I glued magnets directly to the housing, but I think some special thin plastic frame, that will hold magnets properly is strongly needed. We will also have to glue the lens to the focus wheel gear. I used a hot glue gun instead of cyanoacrylate for a test.

Advantages:
> no additional friction!!!
> still cheap and reliable solution

Disadvanteges:
> a bit more complex
> two more gluing points (the lens, the magnet frame)

Have something to think about..
« Last Edit: August 31, 2020, 08:26:13 pm by VGN »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I'm glad to say that PDF schematics of the M-board (the most complex one) and B-board (the most trivial one) are avaliable on my repo!
https://github.com/OVGN/OpenIRV/tree/master/docs/schematics

This is alpha version v0.1.0, I probably will make some changes, but without changelog. First stable version will be v1.0.0.

Looking forward your constructive criticism, bug reports, recommendations, ideas, etc. ;)

In attachments you can see how M-board and B-board looks like now. For M-board I decided to replace JTAG connector, FFC/FPC connector is quite thin and can be easily passed between enclosure parts out of the device for debugging purposes.

I hope other schematics will be finished by the end of this week. I'm going to purchase components for new revision and start porting HyperRAM controller to new FPGA this week. I have a special board from another project that utilized Spartan-7 FPGA and a single HyperRAM IC for testing.
That will be fun)
« Last Edit: August 31, 2020, 11:27:58 pm by VGN »
 

Offline horstmannhid

  • Contributor
  • Posts: 26
  • Country: de
Four 1.5mm height magnets generate enough force, pulling a metal ring, mounted on the lens. I glued magnets directly to the housing, but I think some special thin plastic frame, that will hold magnets properly is strongly needed. We will also have to glue the lens to the focus wheel gear. I used a hot glue gun instead of cyanoacrylate for a test.

That sounds great for this little test. What I meant is using a ring magnet with the right dimensions. They can be bought via ebay. Have a look at the attached picture. If there is a fitting magnet no additional plastic frame for the magnets is needed. Just glue the ring magnet to the housing.


Had a look at the schematic. Debouncing of the buttons will be done in VHDL via a filter in the FPGA? In the microcontroller unit it is an annoying task to debounce something.
 

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
In the original design they seem to have gone to great lengths to keep the lens assembly from rotating when focusing.  There are 4 sliding teflon surfaces between the focusing ring and the lens.  Was that to keep wear particles from the threads off the sensor?

Looks like the foam gasket that seals the bottom of the lens to the housing is not thick enough to seal for close focusing if it requires more than about .5mm expansion from the stock setting. 

A popular way to get around lead screw slop on 3D printers is plastic nuts. They apparently last long enough for that extreme task.  Why not replace the threaded focusing ring with a combined focusing ring + ring gear version.  The SLA printer may be up to the task.  The threads don't have to be perfect, the errors control the slop.  And you get rid of a glue joint !  The threaded portion could be backed with a circlip if required. edit, not enough room
« Last Edit: September 01, 2020, 04:37:05 pm by bicycleguy »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
That sounds great for this little test. What I meant is using a ring magnet with the right dimensions. They can be bought via ebay. Have a look at the attached picture. If there is a fitting magnet no additional plastic frame for the magnets is needed. Just glue the ring magnet to the housing.
Yes, a ring magnet would be the best solution. On the other hand, small magnets are very cheap and easy to find, while ring magnets are quite exotic. Anyway I measured mounting seat dimentions, you can find them in attachments.

Had a look at the schematic. Debouncing of the buttons will be done in VHDL via a filter in the FPGA? In the microcontroller unit it is an annoying task to debounce something.
Buttons pull-ups and debouncing capacitors are at the P-board (peripheral). Of course, there is additional debouncing logic in the HDL design. Agree, filtering buttons by internal mcu is a poor idea.



In the original design they seem to have gone to great lengths to keep the lens assembly from rotating when focusing.  There are 4 sliding teflon surfaces between the focusing ring and the lens.  Was that to keep wear particles from the threads off the sensor?
This unit looks quite over engineered for me. Maybe I don't understand something, but really, no idea, why using so many parts just for a fixed focus...

Looks like the foam gasket that seals the bottom of the lens to the housing is not thick enough to seal for close focusing if it requires more than about .5mm expansion from the stock setting. 
Right, but I don't think that we need this foam gasket, it prevents lens rotation a bit. Also this camera is not supposed to be used in any aggressive environment, no water/dust protection. BTW, even Flir Vue Pro is not water/dust resistant.

A popular way to get around lead screw slop on 3D printers is plastic nuts. They apparently last long enough for that extreme task.  Why not replace the threaded focusing ring with a combined focusing ring + ring gear version.  The SLA printer may be up to the task.  The threads don't have to be perfect, the errors control the slop.  And you get rid of a glue joint !  The threaded portion could be backed with a circlip if required. edit, not enough room
I'm afraid, that custom plastic nut probably will rotate too hard, and this tiny focus motor won't be able to make it move. All the more so I'm going to change the focus motor with higher RPM one. We should somehow keep lens rotating very easily and without backlash, otherwise, there is no chance for fast focus adjustment.
« Last Edit: September 01, 2020, 05:38:26 pm by VGN »
 

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
Here's what I was thinking of:

There is enough room for a steel ring spring that could be tensioned for a nice fit.  I haven't tried on my filament printer.  Wonder how it would do on SLA.
« Last Edit: September 01, 2020, 08:39:11 pm by bicycleguy »
 
The following users thanked this post: VGN

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Here's what I was thinking of:
There is enough room for a steel ring spring that could be tensioned for a nice fit.  I haven't tried on my filament printer.  Wonder how it would do on SLA.
WOW! I take my words back. These cutouts are very interesting!  :-+ :clap:
That definitly should be tested on SLA, pity that I don't have any 3D printer at all. I could purchase it for test, but only later with other parts, which I'm going to fix a little too.

1. How are you going to make the steal ring?
2. What about a special internal border that will help to center and fit the lens? Check out attachments. But I'm not completely sure how to make it properly, this is just a suggestion.
3. So the holding ring thread pitch is 0.508 (20mil)?! :palm: I thought it is metric 0.5mm pitch, though I didn't measure it, not even sure that I could...
« Last Edit: September 01, 2020, 11:17:13 pm by VGN »
 

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
1. How are you going to make the steal ring?
Just bend some music wire around a pipe of a guessed dia.
2. What about a special internal border that will help to center and fit the lens? Check out attachments. But I'm not completely sure how to make it properly, this is just a suggestion.
I think the attachment might have messed up, all I see is what I previously posted.    edit  Whoops I didn't look close enough.  Have to think about this a little more.  I was still planning on using a flattened wave spring, but that doesn't make any sense.
3. So the holding ring thread pitch is 0.508 (20mil)?! :palm: I thought it is metric 0.5mm pitch, though I didn't measure it, not even sure that I could...
I don't know the pitch either, but the dia. is .998 inch so I assume 1 inch with 50thrds/inch.  Could be something else.  So short not sure it matters, especially with plastic threads.

I thought you had an SLA printer for the parts you posted?
« Last Edit: September 02, 2020, 12:28:48 am by bicycleguy »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I thought you had an SLA printer for the parts you posted?
No, I don't have it, I just order printing.



P-board schematics are avaliable: https://github.com/OVGN/OpenIRV/tree/master/docs/schematics
Most of the work is done, I just have to finish rounting and check everything very carefully before ordering new PCBs.
Display and Focus boards schematics will be published in a few days.

M-board is also modified a little. Added ESD protection for JTAG. Also increased sensor IOs number from 18 to 20.
I hope that 20 IO + I2C IO-expander is enough to interface most sensors and even high megapix CMOS digital image sensors like MT9F002 and others, for example.

Finally solved the problem with board stacking. Two special M1.6 SMT board spacers allow properly connect main and peripheral boards without original aluminium housing.
 

Offline bicycleguy

  • Frequent Contributor
  • **
  • Posts: 265
  • Country: us
@VGN  Thinking about the focus.  Your original design just needs a weaker spring.  The spring deflection is proportional to thickness^3.  Maybe the existing spring can be etched in something like pcb copper etch solution to the proper thickness.  Not sure if it has a coating on it.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
@VGN  Thinking about the focus.  Your original design just needs a weaker spring.  The spring deflection is proportional to thickness^3.  Maybe the existing spring can be etched in something like pcb copper etch solution to the proper thickness.  Not sure if it has a coating on it.
I also tend to this. I'm going to try both the stencil spring and the magnet based one.

The spring deflection is proportional to thickness^3.  Maybe the existing spring can be etched in something like pcb copper etch solution to the proper thickness.  Not sure if it has a coating on it.
Not sure if that's going to work. Also far not everyone will be able to repeat this.
 

Offline horstmannhid

  • Contributor
  • Posts: 26
  • Country: de
Yes, a ring magnet would be the best solution. On the other hand, small magnets are very cheap and easy to find, while ring magnets are quite exotic. Anyway I measured mounting seat dimentions, you can find them in attachments.
You are right. I also didn't find a fitting ring magnet. I think it would be a lot better to use normal magnets.


Had a look at your new schematics. Great work!

For debouncing of the buttons I usually use the following circuit. So the bouncing is filtered even when the button is pushed.



Do you know the standby current of the whole system? Button 3 seems to turn on the whole system.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Had a look at your new schematics. Great work!
Thank you! Feel free to ask me any questions! Any suggestions are welcome for discussing.

For debouncing of the buttons I usually use the following circuit. So the bouncing is filtered even when the button is pushed.
Oh...right, pull-up + RC filter looks much better, going to fix this. Thanks!

Do you know the standby current of the whole system?
Of course, 8uA (micro amps).

Button 3 seems to turn on the whole system.
Right, button 3 and also EXT_GPI/PWR_ENA pin at the X9 connector is used to enable the whole system.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Focusing mechanics with magnet spring. Works pretty well, though a bit noisy, as focus board amplifies the motor vibrations, this is going to be fixed.
I'm also planning to replace the motor with higher RPM one, probably we could achieve faster lens movement and less noise.

https://youtu.be/jcN1_7Xi5mU
 

Offline Pantheron

  • Contributor
  • Posts: 10
  • Country: de
Dear VGN,

amazing project!
It clearly shows your dedication in every aspect of the complete project.
The lovely designed Focus mechanism.
The Housing.
The Software and even the PCBs are designed with love!

I really appreciate this.


But i also got some questions - sensor related.

You are reading the sensor out with 60fps?
How low can the clock for reading go, so it i spossible to readout at 10fps? or ist just internally fixed to 60fps?

For what is the CMD Pin on the sensor? 1 Wire? Any information on that?




Thank you.






 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
amazing project!
It clearly shows your dedication in every aspect of the complete project.
The lovely designed Focus mechanism.
The Housing.
The Software and even the PCBs are designed with love!

I really appreciate this.
Hi! Many thanks!

You are reading the sensor out with 60fps?

Yes, the raw 14-bit per pixel data is comming out at 60fps. I left in my previos posts a link to my google drive with a video, captured right from ther sensor.
Here is the link: https://drive.google.com/file/d/1QfK6TBKxTnJjb9-Vjo1R4MsX56F_mhFq/view?usp=sharing
I advise you to download (~180MB) this video before viewing, as not smart youtube downgrades video quality too much.

How low can the clock for reading go, so it i spossible to readout at 10fps? or ist just internally fixed to 60fps?
We don't have and I believe will never have access to datasheets of this core, so I actually don't know that. Original board is forwarding clock at 73.636MHz, I'm using the same value. You will be able to experiment with clock rate, as soon as the HDL-design will be published.

On the other hand, I don't think that you really need it. If you would like to integrate this camera with any other hardware, you can choose USB, HDMI, AV or a low speed interface access over GPIO lines at X9 connector of the P-board. I have plans to implement a SPI interface over GPIO line, so that you can control the camera and readout the video data at any FPS you would like.

For what is the CMD Pin on the sensor? 1 Wire? Any information on that?
Forget about any standard interface with this FPA, everything is proprietary, the CMD interface as well. There are a lot of thing to tell about, I'm preparing a special manual for this. Well, the short answer, this is a 1-bit command line, synchronous to the main clock (73.636MHz). It is used to control the sensor. According to my investigations, a special 22 byte command is sent to the sensor to get each new frame. First two bytes of the command are 0x3F and 0xC0. I'm 99.9% sure, that this is a preamble that helps the core to detect a new command. Why? There is no any other strobe to validate the incomming command, 0xC0 = ~0x3F, very likely to be a preamble. Also, the core does not stream anything if I flip any bit in these two bytes. Other 20 bytes keep some special data to control the sensor. Unfortunatelly, the command fields are not byte aligned, that makes investigations a bit harder. There are a lot of bits that do not affect anything. Also a command for one sensor may not fit another, as themal array parameters are very different from core to core. Though I don't know what all the command fields are doing, I could find some common command pattern that should fit all sensors and a single special field that you can quite easily tune to get the image. I have tested this algorithm on my three FPAs and could make all of them working well. I'm sure that the more people will play with this sensor, the faster we will find out the function of each command field. Another way is to reverse engineer the firmware and get this data from original EEPROM. But, I'm not sure that this is a good idea, as this is an intellectual property of Avtoliv/FLIR.
« Last Edit: September 13, 2020, 11:22:45 pm by VGN »
 

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 55
  • Country: de
Do you think it's possible to have the sensor connected via a shielded cable from about 50cms long to the controller electronics?
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Do you think it's possible to have the sensor connected via a shielded cable from about 50cms long to the controller electronics?
Do you mean the ISC0901B0 sensor?
 

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 55
  • Country: de
Yes, only the sensor in a compact as possible housing.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Yes, only the sensor in a compact as possible housing.
Sensor lines IO standard is LVCMOS 2V5. Data stream is running at @73.636MHz SDR. 50cm is a quite long distance, I'd not say that this is impossible, but will be a bit challenging. You should take in account proper termination of the I/O lines to exclude signal "ringing". I also think that using buffers/repeatters for the five high speed lines, i.e. clock, bias, data_odd, data_even, cmd is also a good idea.

After that keep in mind that the interface between FPGA and sensor is source synchronous, i.e. FPGA is generating the clock and sends it to the sensor. Bias and cmd line are also synchronous to the rising edge of the outgoing clock. At the same time sensor outputs data_odd and data_even signals synchronous to the falling edge of the incomming clock. Signals over PCB usually travels at a speed of 165ps/inch. So, 50cm gives about 3.2ns of delay. At 73.636MHz clock period is about 13.6ns. In ideal conditions (without your expander) we have 13.6ns/2 = 6,78ns of margin before and after the active edge of the clock signal to properly read incomming data at the FPGA side. But in case of additional delay of 3.2ns, the margin after the active clock edge will be lower and probably we will have to compensate this delay for proper operation. Current FPGA is quite powerfull to make such kind of tricks, so yes, this is possbile in certain conditions.

I'm not sure, but making such a long sensor power lines can also be a problem. The noise level of bias and core power rails should be as low as it is possible. This is a matter of experiments.
If it is not a secret, why do you need such a long cable?
« Last Edit: September 17, 2020, 07:16:43 pm by VGN »
 

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 55
  • Country: de
Many thanks for the comprehensive answer.
And no it's no secret at all, I would like to mount the sensor on a very small pan/tilt head on a Trimble UX5 drone.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Many thanks for the comprehensive answer.
And no it's no secret at all, I would like to mount the sensor on a very small pan/tilt head on a Trimble UX5 drone.
It was worth starting with this)) In this case I don't really think that this expander will be reliable enough. At the same time dimentions and weight of the camera is quite critical for small UAVs, that cannot carry this big body camera. But I think I have a solution.

The M-board, as the most expensive part, mostly because of the FPGA, was designed in a way to be fully independent from all other boards, including P-board. That's why for small UAV applications a new special board should be designed. Check out attachments, I just have copypasted some parts from the other boards.

We don't need focusing board, only shutter. So, right now the camera module that you see in the attachments weights 78g. The new P-board will add another 10g, resulting to 90g.

I have small experience with UAVs, and absolutely no any experience with FPV, FPV-cameras, interfaces, etc... Are you still flying on NTSC or PAL?  ;D
That's why I have some question to you and community:

1. What do you think about this addon? Is it worth doing?
2. What about total weight and dimentions?
3. What kind of interfaces FPV pilots use nowadays? Is AV enough? What is the output format (NTSC/PAL/or...)?
4. What else is reasonable to add to the board?
« Last Edit: September 18, 2020, 11:05:38 am by VGN »
 

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 55
  • Country: de
It's amazing how you can design and develop in such a short time projects like this !

But in this case, weight is not my problem, my UAV is equipt with a fixed 36 MP mirrorless full frame camera with a 15mm lens and is normally used in mapping and surveying for making orthomosaics in combination with a very capable GPS system.

My problem only is the limited space on the pan-tilt head that I would like to mount on this UAV, after that I removed the above mentioned camera.
Transmission could be done with HDMI or AV, 2.4GHZ or 5.0GHZ FHSS.
I could opt for a bigger pan-tilt had but i'm afraid this causes to much drag for this fixed wing UAV.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Transmission could be done with HDMI or AV, 2.4GHZ or 5.0GHZ FHSS.
Thanks, streaming HDMI is not a problem, though I can't decide what kind of connector is better to use. I saw different FPV HDMI transmitting hardware, everything looks to be proprietary.

I could opt for a bigger pan-tilt had but i'm afraid this causes to much drag for this fixed wing UAV.
Looks like this drone is not designed to carry any camera outside of the body. But there is some free space at the nose, probabry you could make some mount for a forward looking camera. In any case, the best FPV mode of this themal camera that I can suggest you is 90g within dimention of ~50x50x50mm.
 

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 55
  • Country: de
You are right, normally this drone carries a fixed camera looking straight down.
As this drone is launched from a catapult i can't atach a gimbal to the belly, but i can modify the nose section so it can hold a small pan-tilt head.
Again, weight is not the problem, it's size that matters.
But don't focus to much on my wishes, I can barely wait to get this camera working.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Finally released the first version of HyperBus memory controller. The migration from Spartan-6 was not so easy, as I expected. I had to rework quite a lot of things, and spent much more time, than I planned, but this controller is really one of the most important and critical parts of this project.

I created a special thread about this memory controller. If you have any questions or suggestions, consider using this thread:  https://www.eevblog.com/forum/fpga/openhbmc-open-source-axi4-based-hyperbus-memory-controller/

Link to the repo: https://github.com/OVGN/OpenHBMC
 

Offline zxase258

  • Contributor
  • Posts: 24
  • Country: cn
amazing project!
It clearly shows your dedication in every aspect of the complete project.
The lovely designed Focus mechanism.
The Housing.
The Software and even the PCBs are designed with love!

I really appreciate this.
Hi! Many thanks!

You are reading the sensor out with 60fps?

Yes, the raw 14-bit per pixel data is comming out at 60fps. I left in my previos posts a link to my google drive with a video, captured right from ther sensor.
Here is the link: https://drive.google.com/file/d/1QfK6TBKxTnJjb9-Vjo1R4MsX56F_mhFq/view?usp=sharing
I advise you to download (~180MB) this video before viewing, as not smart youtube downgrades video quality too much.

How low can the clock for reading go, so it i spossible to readout at 10fps? or ist just internally fixed to 60fps?
We don't have and I believe will never have access to datasheets of this core, so I actually don't know that. Original board is forwarding clock at 73.636MHz, I'm using the same value. You will be able to experiment with clock rate, as soon as the HDL-design will be published.

On the other hand, I don't think that you really need it. If you would like to integrate this camera with any other hardware, you can choose USB, HDMI, AV or a low speed interface access over GPIO lines at X9 connector of the P-board. I have plans to implement a SPI interface over GPIO line, so that you can control the camera and readout the video data at any FPS you would like.

For what is the CMD Pin on the sensor? 1 Wire? Any information on that?
Forget about any standard interface with this FPA, everything is proprietary, the CMD interface as well. There are a lot of thing to tell about, I'm preparing a special manual for this. Well, the short answer, this is a 1-bit command line, synchronous to the main clock (73.636MHz). It is used to control the sensor. According to my investigations, a special 22 byte command is sent to the sensor to get each new frame. First two bytes of the command are 0x3F and 0xC0. I'm 99.9% sure, that this is a preamble that helps the core to detect a new command. Why? There is no any other strobe to validate the incomming command, 0xC0 = ~0x3F, very likely to be a preamble. Also, the core does not stream anything if I flip any bit in these two bytes. Other 20 bytes keep some special data to control the sensor. Unfortunatelly, the command fields are not byte aligned, that makes investigations a bit harder. There are a lot of bits that do not affect anything. Also a command for one sensor may not fit another, as themal array parameters are very different from core to core. Though I don't know what all the command fields are doing, I could find some common command pattern that should fit all sensors and a single special field that you can quite easily tune to get the image. I have tested this algorithm on my three FPAs and could make all of them working well. I'm sure that the more people will play with this sensor, the faster we will find out the function of each command field. Another way is to reverse engineer the firmware and get this data from original EEPROM. But, I'm not sure that this is a good idea, as this is an intellectual property of Avtoliv/FLIR.

Hello VGN, in the FPA timing I measured, there is another signal I don't understand its function. This signal is temporarily called "PIN6", and it appears together with the FPA data output PIN2 and PIN3. I don’t know what it does. Is it “exposure/integration” control?
In addition, from the rising edge of FSYNC (DATA IN) to the rising edge of the first data output, there are a total of 200 clocks. I think FPA samples on the falling edge of the clock.
I got a FLIR TAU324, it uses ISC0601B detector, I will measure the timing of ISC0601B. I think I also need an "FPA protocol analyzer". Specifically, I need an FPGA to monitor the commands sent by the camera to the FPA, and then send these commands to the computer to analyze how the camera configures the FPA. However, I am still new to FPGA, maybe I should ask my friend to help me do this. :palm:

73 BI4LBK.
« Last Edit: September 29, 2020, 12:43:24 pm by zxase258 »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hello VGN, in the FPA timing I measured, there is another signal I don't understand its function. This signal is temporarily called "PIN6", and it appears together with the FPA data output PIN2 and PIN3. I don’t know what it does. Is it “exposure/integration” control?
If you are asking about ISC0901B0, your "PIN6" looks like to be a pixel BIAS value, check out schematic of the M-board in my repo. Controller sends 7-bit (in fact only 6-bit are significiant) LSB-first BIAS values for each pixel, each row. Approximate value is about ~0x25.

In addition, from the rising edge of FSYNC (DATA IN) to the rising edge of the first data output, there are a total of 200 clocks. I think FPA samples on the falling edge of the clock.
Is FSYNC (DATA IN)  a CMD lines in my terms? Anyway don't focus at the number of the clocks of the data output relative to the command. There is a parameter in the command that determines this value.

Concerning to the sampling edge, you are right, sensor works at the falling edge. Check out attachments. cmd_to_clk - is command or bias comming out of the controller. data_to_clk is one of the data lines comming out of the sensor.

That would be very cool if you get ISC0601B oscilloscore timing. I think this ISC0601B and ISC0901B0 are very common.
 

Offline tisher

  • Contributor
  • Posts: 11
  • Country: pl
Car/vehicle parts :)

NV0  (flir pathfindir)

(12 pin connector with composite Video-out)
334-0008-00P
334-0005-00S
334-0001-00

NV1 (BMW):

(12 pin connector, no composite Video-out)
66546988779
66549134267
66549141606
66549132753
66546988774
66549125733

NV2:

(6 pin connector)
4H0980552
9205899 (66549205899)
9175668 (66549205899)

NV3:

(4 pin connector)
A2229057307
A2229052805
A2228201597
4G0980552A
9322653 (66549322653)

NV4:
???

btw, wat's lens dimensions in NV3?
« Last Edit: October 18, 2020, 10:31:34 pm by tisher »
 
The following users thanked this post: fest, RO

Offline LesioQ

  • Regular Contributor
  • *
  • Posts: 72
  • Country: pl
  • Every king should be naked.
NV3 lens' clear aperture is 15mm. Objective is of a two-lens design.
Lots of cost-saving on NV3, including Ge protective window thickness being 1/2 NV2.
 

Offline tisher

  • Contributor
  • Posts: 11
  • Country: pl
so 'only' need to find focal length for first lens  :-+
 

Offline mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 14125
  • Country: gb
    • Mike's Electric Stuff
I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.
A few years ago, after I did a teardown video on one of these, someone emailed me, with some info they'd reverse-engineered, but didn't want to publish due to ITAR concerns.
The main point was that they had found that the original FPGA used a Microblaze core, and it was easy to find the code image in the flash memory, and after disassembling with IDA, they managed to reverse-engineer the handshake protocol used to authenticate the camera with the control box.
There might be some useful insights in to how any sensor-unique data is stored.
 
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline Ismsanmar

  • Regular Contributor
  • *
  • Posts: 53
  • Country: es
I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.
A few years ago, after I did a teardown video on one of these, someone emailed me, with some info they'd reverse-engineered, but didn't want to publish due to ITAR concerns.
The main point was that they had found that the original FPGA used a Microblaze core, and it was easy to find the code image in the flash memory, and after disassembling with IDA, they managed to reverse-engineer the handshake protocol used to authenticate the camera with the control box.
There might be some useful insights in to how any sensor-unique data is stored.

Are you talking about this?:
https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/
 

Offline LesioQ

  • Regular Contributor
  • *
  • Posts: 72
  • Country: pl
  • Every king should be naked.
so 'only' need to find focal length for first lens  :-+
What's the purpose of it's focal length ? For f-stop calculation You need effective f.l. of complete objective.
 

Offline mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 14125
  • Country: gb
    • Mike's Electric Stuff

Had a look at the schematic. Debouncing of the buttons will be done in VHDL via a filter in the FPGA? In the microcontroller unit it is an annoying task to debounce something.
Buttons pull-ups and debouncing capacitors are at the P-board (peripheral). Of course, there is additional debouncing logic in the HDL design. Agree, filtering buttons by internal mcu is a poor idea.


Debouncing entirely in software is trivially easy - you simply sample the buttons every 20-40mS or so.

« Last Edit: October 18, 2020, 11:13:27 am by mikeselectricstuff »
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline tmbinc

  • Frequent Contributor
  • **
  • Posts: 253
I would be _really_ curious about firmware dumps of these devices, or any specification that go beyond the two-page summaries.
A few years ago, after I did a teardown video on one of these, someone emailed me, with some info they'd reverse-engineered, but didn't want to publish due to ITAR concerns.
The main point was that they had found that the original FPGA used a Microblaze core, and it was easy to find the code image in the flash memory, and after disassembling with IDA, they managed to reverse-engineer the handshake protocol used to authenticate the camera with the control box.
There might be some useful insights in to how any sensor-unique data is stored.
Hi Mike, yes, that was me. I've since then published my findings - in slightly redacted form - in https://www.eevblog.com/forum/thermal-imaging/autoliv-nv2-on-raspberry-pi/.

However, my question specifically is a.) about the NV3 family, and b.) more specifcally the FLIR PathFindIR II, and how it differs from the automotive NV3 that we've all seen.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hello everyone, some updates...

Sorry again for this delay... but this 2020 year is a nightmare in absolutely every aspect of life.
But I try to do my best to keep developing the project no matter what happens.

Back on project:

1. As the hardware development is almost done, I have ordered electronic parts for the new hardware revision last monday, hope to get them in 2-3 weeks (yes...too long, but there is no any other way to speed up this process).

2. I have finally finished all shematics and routed all PCBs, including recomendations that I received from you. The whole set of schematics is published, chekout my repo.
PCB board production time is about 1 week, but I'm going to ordered new boards next week to get both boards and components simultaneously. If you have any suggestions according to newest schematics, please let me know, this week is the last chance to make any changes.
One more link to the repo: https://github.com/OVGN/OpenIRV/tree/master/docs/schematics

3. All firmware and HDL sources will be published along with porting to new hardware. As soon as I get sure that new hardware is working well, I will start beta production for testers and developers.
« Last Edit: October 23, 2020, 07:15:48 pm by VGN »
 

Offline dmendesf

  • Frequent Contributor
  • **
  • Posts: 342
  • Country: br
Congratulations, your Altium skills are top notch.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hi! Some updates.

Good news, I received most part of active components, i.e. power supply ICs and the FPGA! Now waiting for some passive smd stuff from another supplier. New PCBs were ordered several days ago. Hope to start assembling everything in a week or a little later.

Meanwhile, I decided to make some improvements to the focusing system. First, I have modified the motor mount. Earlier I noticed that motor rigid connection to the board is not a good idea, as motor vibrations sound is amplified by the focus PCB. I decided to use a special silicone cover between the motor and the motor mount. I haven't printed this parts yet, so let me know if you have ideas of impromevents.

The second modification is the focus motor itself. Somebody noticed that focus adjustment is really slow. Yes, this is true. This is because I have selected a motor with very high gear ration and low rpm (1:136 @240RPM) as I was not sure if this tiny motor can apply enough force to move the lens. There is another type of this motor with lower gear ration and higher RPM (1:26 @1200RPM). I haven't ordered it yet, but...I decided to modify one of my motors. If you take it apart, you will find 3 stages of tiny planetary gear. I removed one the stages, shortened the motor housing and put all back together. Now it is running much faster! Checkout the difference at video:
https://youtu.be/GuIjXj8-yEI

Congratulations, your Altium skills are top notch.
Thanks, though there so many thing to learn)

« Last Edit: November 07, 2020, 02:53:06 am by VGN »
 
The following users thanked this post: ajw107

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 55
  • Country: de
New focus speed is for my purpose now fast enough.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
New focus speed is for my purpose now fast enough.
One more video of focusing right from the thermal camera. This is not autofocus yet, I'm adjusting the focus manually and not accurate enough sometimes. The autofocus will do this job much better that me.

Youtube collapces video quality too much, that's why download this video before viewing, it is only ~39MB.
https://drive.google.com/file/d/1_MNzznl6It1Ix80HIM0NVI9G4iD7iLla/view?usp=sharing
« Last Edit: November 07, 2020, 12:40:54 pm by VGN »
 
The following users thanked this post: ddrl46, Cat, rockwell, horstmannhid

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
While waiting for PCBs and components, I decided to reveal some future plans.

ISC0901 is a pretty good 336x256 60FPS sensor. This characteristics is enougth for quite a lot of different applications.
But I have one a bit crazy idea. I would like to significially increase the resolution. As you know, the whole hardware was
designed in a way to support any other sensor (I mean, configurable power lines, a lot of high performance IOs, RAM, etc.)
But unfortunatelly this is really hard to find cheap high resolution sensor...

If you look at the SEM photos of LWIR array, you could see a quite interesting pixel structure. Note, this is a photo of the ISC0601, not ISC0901.
The main difference is the pixel pitch. ISC0601 has 25um pixel pitch, while ISC0901's pixel pitch looks like to be 17um.
I could not find SEM photos of the ISC0901 sensor pixel array, except one photo of the beam holding "via", that proves somehow that ISC0901 has the same
pixel pattern as ISC0601.

If you have or saw somewhere the SEM photos of ISC0901 pixel array, please share it here, that will be very helpful!

The pink area is actually the active pixel, this is a resistor that is supported by two special thin beams, that allow to make
thermal insulation of the pixel from the die and other pixels. The radiation comming from the lens heats up each pixel.
The resistance of the each pixel changes according to the radiation level. This resistance value is calculated by internal ADC,
so that's how we get the pixel value.

Due to technology limitations, the active pixel area is only about 21% of the whole FPA area. There are huge gapes between pixels, that are not doing any useful work.
So simply shifting the array for a 1/2 of pixel pitch value in X and Y direction, we could increase the resolution from 336x256 to 672x512.

But this is also not a limit yet, as making more accurate steps (1/4 or 1/8 pixel pitch) along with special deconvolution algorithms it is possible
to increase the resolution even more! The last attached photo shows the result of x5 super resolution image reconstruction, based on 20 low resolution frames. Looks quite impressive for me.

I'm not sure, but I hope to get a megapixel camera this way))
« Last Edit: November 14, 2020, 03:28:01 pm by VGN »
 
The following users thanked this post: ajw107

Offline zrq

  • Frequent Contributor
  • **
  • Posts: 343
  • Country: 00
If someone in the EU have a faulty sensor, please send it to me. I'll be happy to get some SEM images. (I'll try my best  ;) to persuade my advisor to allow me to do that)
 
The following users thanked this post: VGN

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
If someone in the EU have a faulty sensor, please send it to me. I'll be happy to get some SEM images. (I'll try my best  ;) to persuade my advisor to allow me to do that)
That will be really nice to get SEM photos, but not mandatory, as we know the most important value - the pixel pitch of 17um. FLIR’s Tau2 camera core has 17 μm pixel pitch and we know that it is based on ISC0901.

Probably knowing dimentions of the pixel's active area and gaps around it will be helpful for proper enlarged image calculation. I think this the only reason to sacrify single lwir core and get SEM photos.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Now two words about how I'm going to implement this pixel shifting.

Well, we are talking about micrometer precision level of displacement. I know that there is a trick of using handheld motion effect,
that allows to get this small sensor displacements (Fluke implemented this feature on some of their thermal imagers).
But don't like it at all, as handheld motion based super resolution (SR) is not accurate enough and also will not work in case of tripod mounting.

I'm going to use piezo actuators along with a custom XY-stage.

You can find a lot of info about piezo actuators and the physics of  inverse piezoelectric effect. In two words this is crystal that can enlarge
for some small value if you apply some voltage. Piezo actuator behaves like a common capcacitor, and displacement value is equivalent to the charge level.
The dependency between the voltage and actual displacenent value is linear. It is possible to get even nm accuracy if desired.

ISC0901 has 17um pixel pitch, so we should select an actuator with a displacement value > pixel_pitch/2. I have selected AE0203D18H18DF, that has
max displacement level of 19.0 ±2.0um at 150V. Actuator dimentions are 18x3x2mm. The generated displacement force just blows my mind, it is about 200N (yes, two hundred).

I made some sketches of the XY stage and also some simulations. The thickness of the moving platform beams is about 0.8mm. If made of 6061 aluminium we need to
apply only 10N to displace the platform for about 16um, while the actuator can generate 200N.

Note that the movement level at attached gifs is amplified a lot, this is not the real value, as 18um displacement value is so small, that this is simply impossible to see it with an unaided eye.

I have never developed anything using such kind of XY-stages and piezo actuators, so suggestions and critics are welcome.



Advantages:

1. It is possible to significially increase the resolution. The gaps between active pixel area are so big, that pitch/2 shift will give a clear x4 resolution. More accurate shifting steps along with deconvolution image processing will increase the resolution even more, I hope to get a megapixel.
2. The construction will be fully backward compatible with whole other design, including focusing board.
3. This XY stage can be used with any other sensors, if attached over special adapter. New LWIR cores have even lower pixel pitch of 12um and we can still increase the resolution.

Disadvantages:

1. The piezo actuator, this piece of... shi ceramics... costs around 100$. Yes, it is even more expensive that the FPGA. And...we need two of them for X and Y axis.  :scared:
2. We will have to deal with high DC voltages under 150V, well this is a bit more dangerous than usual. There are special ICs for driving actuators with integrated dc-dc boosters that allow to get this high voltage.
3. Not sure, but probably the whole construction will be a bit more fragile, expecially because of the ceramic actuators, though I'm sure you wouldn't let it fall down)
« Last Edit: November 15, 2020, 01:51:13 pm by VGN »
 

Offline Vipitis

  • Frequent Contributor
  • **
  • Posts: 880
  • Country: de
  • aspiring thermal photography enthusiast
Pixel shifting can be found in Sony cameras and Hasselblad. There are many thread in this subforum that touch on the idea and some did tests. Pixel shifting is not viable for video, but a slow photomode for high resolution stills would be a great feature I would love to have.
The claims about active area vs Pixel pitch Sounds a little odd. We know that Leonardo/DRS has a special pixel design that is a lot more area efficient than the one shown here, using "micro umbrellas". It might be the sole reason why they managed to 10μm somewhat useable.

Also a thought: while you shift the sensor, why not include a whole 5 axis platform for in body stabilizion? Sony uses that for their pixel shifting as well. And they do capture like 12 individual frames on the a7RIV to get 120MP and no interpolation.

Do you hope to shift by perfectly 1/2 the pixel pitch or would 1.5 also work?  If you go beyond 4x this would require getting 0.33,0.66 or even 0.25, -0.25 too. While it might seem that perfect precision makes this sound possible - the real world has limitations in holding perfectly still. So a potential photomode could also include stacking of same pixels shifted by 1 to reduce noise.


There is great development done in this technology for surveillance satellites, so it's very much inaccessible.
 

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
In case this paper is of use/interest.....

Vibration induced Resonance and the effects of G forces in microbolometer pixels ......

https://www.jvejournals.com/article/15829/pdf

Fraser
« Last Edit: November 15, 2020, 03:11:41 pm by Fraser »
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 
The following users thanked this post: zrq, VGN

Offline mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 14125
  • Country: gb
    • Mike's Electric Stuff
So simply shifting the array for a 1/2 of pixel pitch value in X and Y direction, we could increase the resolution from 336x256 to 672x512.
Bear in mind you may need a much better ( more expensive) lens to get improved resolution. If it is the case that it's only sampling a fairly small percentage of the sensor area, then you would see noticeable aliasing artifacts (e.g. jaggies on straight edges) unless the existing optics system has been designed to slightly blur the image to antialias it. It may also be the case that there is some post-processing going on as well to reduce aliasing effects.
I'm sure there is some scope to improve resolution by moving the sensor and applying some clever maths, but I'm not convinced the results would be spectacular.
You can see this effect in this video where I was under-sampling a video camera :
https://youtu.be/rQYByorpoFk?t=869
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 
The following users thanked this post: VGN

Offline SilverSolder

  • Super Contributor
  • ***
  • Posts: 6126
  • Country: 00

Wouldn't it have to be a pretty terrible lens to not be able to resolve  336x256, for example?
 

Offline mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 14125
  • Country: gb
    • Mike's Electric Stuff

Wouldn't it have to be a pretty terrible lens to not be able to resolve  336x256, for example?
Yes, but it is probably engineered to that resolution, giving just the right amount of blur to avoid aliasing artifacts at that resolution
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 
The following users thanked this post: SilverSolder

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Pixel shifting can be found in Sony cameras and Hasselblad. There are many thread in this subforum that touch on the idea and some did tests. Pixel shifting is not viable for video, but a slow photomode for high resolution stills would be a great feature I would love to have.

Yes, but as I know pixel shifting in common cameras is mostly used to breaktrough the limitation on Bayer filter, i.e. shifting the sensor you can get true R,G,B values for each pixel, so there is no need to make debayering, as we already have all color components. I don't really know, why it is called super resolution, as it is not that, we don't get more pixels:

Pixel shifting is not viable for video, but a slow photomode for high resolution stills would be a great feature I would love to have.
True, but piezo actuators are extremely fast. So theoretically it is possible to shoot a video, though the FPS will be n^2 times lower, where n is a factor of resolution enlargement.

The claims about active area vs Pixel pitch Sounds a little odd. We know that Leonardo/DRS has a special pixel design that is a lot more area efficient than the one shown here, using "micro umbrellas". It might be the sole reason why they managed to 10μm somewhat useable.
That was news to me! So it turns out that each "micro umbrella" just heats up the same VOx film, which is placed under it, supported by the same know thermal isolation pattern? And we have almost no pixel gaps. This is really incredible! Checkout attachments.

Also a thought: while you shift the sensor, why not include a whole 5 axis platform for in body stabilizion? Sony uses that for their pixel shifting as well. And they do capture like 12 individual frames on the a7RIV to get 120MP and no interpolation.
That is going to be too complex and expensive...Moreover, no idea what to do with 5 axis.

Do you hope to shift by perfectly 1/2 the pixel pitch or would 1.5 also work?  If you go beyond 4x this would require getting 0.33,0.66 or even 0.25, -0.25 too. While it might seem that perfect precision makes this sound possible - the real world has limitations in holding perfectly still. So a potential photomode could also include stacking of same pixels shifted by 1 to reduce noise.

I hope to shift with much smaller steps that 1/2. Anyway this is a matter of real experiments, as I expect some limitations and problems. I also should find some way for step calibration.
« Last Edit: November 15, 2020, 04:02:36 pm by VGN »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Vibration induced Resonance and the effects of G forces in microbolometer pixels ......
Thanks, Fraser. That should be taken in account, but I think that vibration frequency caused by pixel shifting actuators will be far away from the sensor's pixel resonance frequency.

Bear in mind you may need a much better ( more expensive) lens to get improved resolution.
Thanks, Mike, I didn't thought about that. Well, that may be a huge problem. I'm not really good with optics, but original FLIR E8 lens is much worse that this one, but it was also designed for the right same ISC0901 sensor.

Do you have any ideas how to check this lens capabilities?
 

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
1. The piezo actuator, this piece of... shi ceramics... costs around 100$. Yes, it is even more expensive that the FPGA. And...we need two of them for X and Y axis.  :scared:

I would give these a try: https://nl.aliexpress.com/item/32874018830.html?spm=a2g0o.productlist.0.0.2b033905e3vloB&algo_pvid=23e2f0b8-e665-4619-bd80-5c8e64b3aa5c&algo_expid=23e2f0b8-e665-4619-bd80-5c8e64b3aa5c-11&btsid=0b0a555c16054735796882807e75f6&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_

180µm for 100V, should be around 18µm for 10V, and as you are 3D printing your housing and absolute accuracy is not required for this application, I think these would be sufficient.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I would give these a try: ...
180µm for 100V, should be around 18µm for 10V, and as you are 3D printing your housing and absolute accuracy is not required for this application, I think these would be sufficient.
Hm... the characteristics look quite fantastic for this dimentions. Have you ever used this parts?

For example, both different well known manufacturers, and the parts with common dimentions have commom characteristics:
Kemet's AE0203D18H18DF from here: https://ru.mouser.com/datasheet/2/212/1/KEM_P0101_AE-1518874.pdf
PI's P-882.51 from here: https://static.pi-usa.us/fileadmin/user_upload/physik_instrumente/files/datasheets/P-882-Datasheet.pdf

I have absolutely no experience with piezo actuators, but chinese ones look really strange. Maybe I'm wrong, not sure this is truth. We need anyone how can prove that.
 

Offline Bill W

  • Super Contributor
  • ***
  • Posts: 1141
  • Country: gb
    • Fire TICS

Wouldn't it have to be a pretty terrible lens to not be able to resolve  336x256, for example?

Not at all.  The real issue is the lens 'circle of confusion' versus the pixel pitch.  Lots of appropriate maths at
https://www.dofmaster.com/dofjs.html

Our thermal lenses score badly due to the large apertures used and the expense of processing or multiple elements.
A simple 2 element 'spherical' design (50mm f/0.7) meant for a Pevicon tube really does not resolve well on 17µm pitch.

Bill
 
The following users thanked this post: SilverSolder

Offline bap2703

  • Regular Contributor
  • *
  • Posts: 202
  • Country: io
Do you have any ideas how to check this lens capabilities?

At first you can probably do a simple check:
Can you get a sharp image of a point or edge source?

The real thing is called measuring the modulation transfer function (MTF) of the lens.
Basically you describe an image in terms of spatial frequencies: how fast the luminosity is changing quickly across pixels.
Details equal to high frequencies for example. Since you played with frequency domain filtering of images you'll quickly grasp how that works.
Think of lenses like low-pass filters: in order to get more details, you need to grab more high-frequencies.
Lens designers would probably match the cut-off to the targeted sensor pitch if it allows to build it cheaper.

You can probably get to half the pitch without too much trouble (or software sharpening), but more than that you'll probably notice the lens limitations.

You can read more there:
https://www.umicore.com/en/newsroom/the-evolution-of-lens-designs-for-12micron-uncooled-lwir-detectors/

 
The following users thanked this post: VGN

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
I would give these a try: ...
180µm for 100V, should be around 18µm for 10V, and as you are 3D printing your housing and absolute accuracy is not required for this application, I think these would be sufficient.
Hm... the characteristics look quite fantastic for this dimentions. Have you ever used this parts?

For example, both different well known manufacturers, and the parts with common dimentions have commom characteristics:
Kemet's AE0203D18H18DF from here: https://ru.mouser.com/datasheet/2/212/1/KEM_P0101_AE-1518874.pdf
PI's P-882.51 from here: https://static.pi-usa.us/fileadmin/user_upload/physik_instrumente/files/datasheets/P-882-Datasheet.pdf

I have absolutely no experience with piezo actuators, but chinese ones look really strange. Maybe I'm wrong, not sure this is truth. We need anyone how can prove that.

I have no experience with those yet. I did an experiment with a low cost piezo buzzer (with the intention of building an Scanning Fabry-Perot Interferometer like http://repairfaq.cis.upenn.edu/Misc/sale/sfpiins1.htm).

This is the type of piezo I am talking about: https://nl.aliexpress.com/item/4000120679339.html?spm=a2g0o.productlist.0.0.5cde25c3H0l9jK&algo_pvid=39a8075f-6e25-492a-911d-34217e99d7f2&algo_expid=39a8075f-6e25-492a-911d-34217e99d7f2-3&btsid=0bb0623416055593233587628ed6a4&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_

These have a ceramic layer of only approximately 0.2mm thick. It was connected directly to my signal generator with a 10Vpp output. A mirror was glued to the piezo, and distance was measured with a DIY interferometer (details here: http://www.repairfaq.org/sam/uMD1/). This gave me a little less than 250nm pp. Taking into account the ceramic material was only 0.2mm thick, this would mean 6.25µm pp for a 5mm thick material. This is below the 18µm spec for 10V, but if similar specs, with a higher voltage, 17µm must be easily achievable.

To be sure, I also ordered a set (https://nl.aliexpress.com/item/32952294614.html?spm=a2g0o.productlist.0.0.4da739052Pa8qc&algo_pvid=aa231da8-f9e6-4c2d-97a4-be556ee5db3d&algo_expid=aa231da8-f9e6-4c2d-97a4-be556ee5db3d-0&btsid=0bb0624516055606928738744e86ab&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_) and will test them the same way.
 
 

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
The results above need to be divided by 2, as I used a plane mirror interferometer setup instead of my "usual" linear setup with retro reflector, but did not change the settings accordingly in µmd1.

So this means with a 10V signal that kind of piezo only does 125nm instead of the 250nm shown in the result plot.
 

Offline LesioQ

  • Regular Contributor
  • *
  • Posts: 72
  • Country: pl
  • Every king should be naked.
The ultimate limit will be Airy disk dia, i.e. physics.

http://www.calctool.org/CALC/phys/optics/spot_size
« Last Edit: November 17, 2020, 10:17:13 am by LesioQ »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
At first you can probably do a simple check:
Can you get a sharp image of a point or edge source?
Sure.

Experiment #1:
The distance to the object is about 5m. I used an aluminium tape to make a sharp edge. I was playing with focus and camera position for ~20 minutes. This is the best edge sharpness, that I could achieve. I started to feel lens limitations)
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
At first you can probably do a simple check:
Can you get a sharp image of a point or edge source?

Experiment #2:
The distance to the object is about 0,4m:
« Last Edit: November 17, 2020, 11:51:10 am by VGN »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
The real thing is called measuring the modulation transfer function (MTF) of the lens.
Basically you describe an image in terms of spatial frequencies: how fast the luminosity is changing quickly across pixels.
Details equal to high frequencies for example. Since you played with frequency domain filtering of images you'll quickly grasp how that works.
Think of lenses like low-pass filters: in order to get more details, you need to grab more high-frequencies.
Lens designers would probably match the cut-off to the targeted sensor pitch if it allows to build it cheaper.

bap2703, thank you very much, that was really helpful! Looks like I started somehow understand what is going on...  ::)
So, it turn out that using pixel shifting I just actually decrease the sensor pixel pitch. And if I want to keep the image sharpness, I should increase the lens "bandwidth" (as it is an optical low-pass filter)

Quote
From the article:
Since the lens performance is essentially diffraction limited, there is only one way to increase the MTF of the lens so that it remains constant for the higher spatial frequency. That is to make the lens faster in the ratio of the change in pixel pitch.
Ok, so if I got it right, we need a faster lens if we want to decrease the pixel pitch. Probably, first we should determine the lens f-number.

You can probably get to half the pitch without too much trouble (or software sharpening), but more than that you'll probably notice the lens limitations.
If I'm not wrong, not only lens limitations. As the pixel itself is not a spot, i.e. it has some dimentions, we will also face with pixel overlap at very small steps.
« Last Edit: November 17, 2020, 11:54:46 am by VGN »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I have no experience with those yet. I did an experiment with a low cost piezo buzzer (with the intention of building an Scanning Fabry-Perot Interferometer like http://repairfaq.cis.upenn.edu/Misc/sale/sfpiins1.htm).

This is the type of piezo I am talking about: https://nl.aliexpress.com/item/4000120679339.html?spm=a2g0o.productlist.0.0.5cde25c3H0l9jK&algo_pvid=39a8075f-6e25-492a-911d-34217e99d7f2&algo_expid=39a8075f-6e25-492a-911d-34217e99d7f2-3&btsid=0bb0623416055593233587628ed6a4&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_

These have a ceramic layer of only approximately 0.2mm thick. It was connected directly to my signal generator with a 10Vpp output. A mirror was glued to the piezo, and distance was measured with a DIY interferometer (details here: http://www.repairfaq.org/sam/uMD1/). This gave me a little less than 250nm pp. Taking into account the ceramic material was only 0.2mm thick, this would mean 6.25µm pp for a 5mm thick material. This is below the 18µm spec for 10V, but if similar specs, with a higher voltage, 17µm must be easily achievable.

To be sure, I also ordered a set (https://nl.aliexpress.com/item/32952294614.html?spm=a2g0o.productlist.0.0.4da739052Pa8qc&algo_pvid=aa231da8-f9e6-4c2d-97a4-be556ee5db3d&algo_expid=aa231da8-f9e6-4c2d-97a4-be556ee5db3d-0&btsid=0bb0624516055606928738744e86ab&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_) and will test them the same way.

Nice project!

This gave me a little less than 250nm pp. Taking into account the ceramic material was only 0.2mm thick, this would mean 6.25µm pp for a 5mm thick material. This is below the 18µm spec for 10V, but if similar specs, with a higher voltage, 17µm must be easily achievable.
Agree, that makes sense.



Found this "datasheet" for AL1.65X1.65X5D-4F, according to it only 3,8um@90V :-//
https://img.alicdn.com/imgextra/i2/71977092/TB2wz4ybNaK.eBjSZFAXXczFXXa_!!71977092.png

To be sure, I also ordered a set and will test them the same way.
Anyway, I will be very grateful if you share your results with us!
« Last Edit: November 17, 2020, 12:44:36 pm by VGN »
 

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
Just for information, the Autoliv camera is basically a variation on the TAU and TAU 2. Specifications for the various lens options are available. In my experience modern lenses for the TAU cameras are all of adequate resolution to work well with a VGA 17um microbolometer. If you look at suppliers, such as Ophir, it is clear that modern thermal imaging lenses are more than capable of VGA resolutions on 17um pixels and even XGA on 12um pixels.

I would be very surprised if the lens installed in a TAU, or TAU based core was not capable of at least VGA on 17um pixels. Lens manufacturing has become finely honed to offer affordable thermal imaging lenses capable of good resolution. Even Chalcogenide IR Glass lenses can achieve such good performance these days provided the lens designer does his job correctly. A manufacturer will normally design a lens set that will work with a large range of microbolometers and the Ophir catalogue shows this. XGA 12um capability is now common. I attach a link to the TAU brochure and the Ophir catalogue for information. How I would love to have free access to the Ophir range of lenses  ;D. Take a look at the 65221 19mm lens specification on page 52. That is not a super expensive lens yet it is adequate resolution for 1024 x 768 12um pixels. That 19mm lens is used in the ThermApp camera.

http://52ebad10ee97eea25d5e-d7d40819259e7d3022d9ad53e3694148.r84.cf3.rackcdn.com/UK_Z24-FLIR_Tau2_Family_Brochure_DS_CAT-08.pdf

http://52ebad10ee97eea25d5e-d7d40819259e7d3022d9ad53e3694148.r84.cf3.rackcdn.com/UK_Z24-FLIR_Tau2_Family_Brochure_DS_CAT-08.pdf

From my perspective, I just cannot see a modern thermal imaging lens manufacturer going to the effort of making a poorly performing lens that cannot cope with VGA 17um pixels as a minimum. Where the mechanical limits permit, I would expect most modern lenses to cope well with XGA. It is true, as Bill_W has stated, that older thermal imaging cameras that used much larger pixels and arrays that were QVGA or smaller did have resolution limits associated with the lens block. That was an era when pure Germanium lenses were cut on mono point diamond lens forming lathes. Such was a very expensive process and remains so to this day. The lenses tended to be cut to the requirement rather than to needlessly exceed it at additional cost. Times have changed with the introduction of Chalcogenide IR glass lenses that are precision moulded and more advanced pure Germanium cutting processes. Thermal camera lens manufacturing has had to advance to keep in step with newer, smaller and higher resolution imaging sensors.

I caveat all of the above by saying that a manufacturer who wants a camera built down to a certain production cost may still cut corners on the lens part of the design. It is still possible to have relatively poorly performing thermal imaging lenses produced where image quality is not a high priority. Such lenses are commonly made for less demanding applications such as industrial IR temperature sensors so lens manufacturers do have relatively low grade lens production facilities.

FLIR went as far as using a ‘printed’ diffractive silicon lens in their Lepton core and that lens is resolution limited. How I wish there was a conventional lens option for that core.

Fraser
« Last Edit: November 17, 2020, 01:24:40 pm by Fraser »
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 
The following users thanked this post: VGN

Offline bap2703

  • Regular Contributor
  • *
  • Posts: 202
  • Country: io
Experiment #2:
The distance to the object is about 0,4m:

I'd say you lens looks definitely adequate to take advantage of superresolution.

If I'm not wrong, not only lens limitations. As the pixel itself is not a spot, i.e. it has some dimentions, we will also face with pixel overlap at very small steps.

You are right and that's mathematically all the same:
- the response of the lens is an image where every source point is spread (and that's called the PSF, point spread function)
- a pixel has a spatial response too, be it from it's physical size or some "antennae" effect (you posted picture of pixels with a grid structure for example)

When you know them, you can to some extent compensate their effects.
That's called deconvolution and it's used everywhere.
 
The following users thanked this post: VGN

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
Found this "datasheet" for AL1.65X1.65X5D-4F, according to it only 3,8um@90V :-//
https://img.alicdn.com/imgextra/i2/71977092/TB2wz4ybNaK.eBjSZFAXXczFXXa_!!71977092.png

 :o :-DD? That could be correct also of coarse! Anyway, it will be fun to play with, and I need to "use" my interferometer...

Found this "datasheet" for AL1.65X1.65X5D-4F, according to it only 3,8um@90V :-//
https://img.alicdn.com/imgextra/i2/71977092/TB2wz4ybNaK.eBjSZFAXXczFXXa_!!71977092.png
Anyway, I will be very grateful if you share your results with us!
 
Will do, but as we all now, it can take a while until the Chinese Santa Clause comes to our door...
« Last Edit: November 17, 2020, 07:26:41 pm by _Wim_ »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I'd say you lens looks definitely adequate to take advantage of superresolution.
I hope that very much too. But I also want to try to prove this theoretically if it is possible.

The ultimate limit will be Airy disk dia, i.e. physics.

http://www.calctool.org/CALC/phys/optics/spot_size
Ok, lets calculate this Airy disk dia.

First, let's get the f-number of the lens.
According to this datasheet (table 6.1): http://www.safetyvision.com/sites/safetyvision.com/files/FLIR_PathFindIRII_User_Guide_1.pdf
The lens focal length is f = 19mm (seems to be true). The aperture circle dia D = 15mm. In this way f-number = f/D = 19/15 = 1.26(6). The closest standard value is 1.25.
So, let's say that we have a f/1.25 lens.

Finally for LWIR spectral band of (8 - 14um) we have an Airy disk dia of (21.35 - 42.7um). Well...even smallest value of this range is much larger than the actual pixel size for a 17um pitch technology...

Again, I'm not good enough with optics, but the result looks a bit weird for me, I didn't expect such values. Any ideas what am I doing or interpreting wrong?  |O
« Last Edit: November 19, 2020, 01:15:01 pm by VGN »
 

Offline Fraser

  • Super Contributor
  • ***
  • Posts: 13554
  • Country: gb
Your calculations look correct to me.

I did not include the link to the Ophir catalogue in my earlier post, sorry.

Look at the 19mm lens on page 52 and enter its specs into an on line Airy Disk calculator (as I did). It verifies your calculated result.

https://applied-infrared.com.au/images/pdf/Ophir_Thermal_Imaging_Lenses_Catalog.pdf

I believe you are showing the limitations of the optics used in thermal imaging when small pixel sizes are involved. This may go some way to explain why microbolometer arrays that use 12um pixels struggle with contrast.

I believe anti-aliasing and dynamic contrast enhancement techniques are employed to improve the contrast in images produced by thermal cameras. It would appear that any core that uses a pixel size smaller than 25um is battling the Airy Disk limits of the commonly available optical systems.

Thank you for highlighting this situation as I have not previously looked at it for thermal lenses  :-+

Airy Disk Calculator :

http://www.calctool.org/CALC/phys/optics/f_NA

For the referenced 19mm Ophir lens the Airy disk is 27um at 10um wavelength.

Fraser
« Last Edit: November 19, 2020, 06:39:50 pm by Fraser »
If I have helped you please consider a donation : https://gofund.me/c86b0a2c
 

Offline bap2703

  • Regular Contributor
  • *
  • Posts: 202
  • Country: io
The game doesn't end with calculating the airy disc diameter ! That's a common mistake.

What do you want to do next?
--> sample it at more than just one pixel per airy diameter!
Because you're not really interested in sampling the airy disc with one pixel, but sampling the tiny contrast (in amplitude AND space) resulting from merging two close diffracted spots.

You can sample at a pitch much less than the airy diameter.

I found that document that appears to explains it quite well (I had a quick glimpse at it but it seems good).
https://www.researchgate.net/publication/336775082_Resolving_some_spatial_resolution_issues_-_Part_2_When_diffraction_takes_over

That one says a pitch of 1/6 of the airy diameter !


 
The following users thanked this post: Fraser, zrq, VGN, gsumner

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Thank you for highlighting this situation as I have not previously looked at it for thermal lenses  :-+
Thanks to LesioQ and our cruel, but beautiful universe!  ;)



The game doesn't end with calculating the airy disc diameter ! That's a common mistake.

What do you want to do next?
--> sample it at more than just one pixel per airy diameter!
Because you're not really interested in sampling the airy disc with one pixel, but sampling the tiny contrast (in amplitude AND space) resulting from merging two close diffracted spots.

You can sample at a pitch much less than the airy diameter.

I found that document that appears to explains it quite well (I had a quick glimpse at it but it seems good).
First, thank you for this article, it is really mind changing. I recommend everyone to read it.
Second, you are absolutely right, we can sample at a small pitch to get maximum possible spatial resolution that current lens is capable to provide. I found quite nice picture from one astronomy forum, that shows the advantage of high accurate airy disk sampling.

That one says a pitch of 1/6 of the airy diameter !
Here is a quote from the article, explaining why we are talking about 1/6 of the airy diameter for those, who didn't find it:
Quote
Since we know from the Rayleigh criterion that Δxmin equals the radius of the Airy disc, it follows that the best
‘match’ between sensor characteristics and the optics is achieved when the detector pitch p equals one‐third of
the Airy disc radius. In other words: three pixels suffice to resolve the Airy disc radius entirely. The reasoning
then goes that above that threshold, the optics diffraction limitation kicks in. So digitising the radius of the PSF’s
central part with more than three pixels does not improve the final spatial resolution of the image because the
detector only resolves diffraction blur at that point.



BTW, I didn't now, but FLIR has a special UltraMax technology, that allows to increase the resolution. It is very silimilar to Fluke's superresolution.
The most interesting moment is at 0:51, the main idea is the same, just sample the data in gaps between pixels, as active pixel area is really small.
Full video: https://youtu.be/8y2lV3nIhvc
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Some updates.

Today received new two layer PCBs, i.e. buttons, display and the magnets holding ring! Going to assembly this boards today.

Unfortunately looks like due to covid-disaster manufacturer delays boards release, that's why main and peripheral boards will apear in my hands only at next week.
« Last Edit: November 25, 2020, 03:38:30 pm by VGN »
 

Offline Max Planck

  • Regular Contributor
  • *
  • Posts: 97
  • Country: pl
I'd say you lens looks definitely adequate to take advantage of superresolution.
I hope that very much too. But I also want to try to prove this theoretically if it is possible.

The ultimate limit will be Airy disk dia, i.e. physics.

http://www.calctool.org/CALC/phys/optics/spot_size
Ok, lets calculate this Airy disk dia.

First, let's get the f-number of the lens.
According to this datasheet (table 6.1): http://www.safetyvision.com/sites/safetyvision.com/files/FLIR_PathFindIRII_User_Guide_1.pdf
The lens focal length is f = 19mm (seems to be true). The aperture circle dia D = 15mm. In this way f-number = f/D = 19/15 = 1.26(6). The closest standard value is 1.25.
So, let's say that we have a f/1.25 lens.

Finally for LWIR spectral band of (8 - 14um) we have an Airy disk dia of (21.35 - 42.7um). Well...even smallest value of this range is much larger than the actual pixel size for a 17um pitch technology...

Again, I'm not good enough with optics, but the result looks a bit weird for me, I didn't expect such values. Any ideas what am I doing or interpreting wrong?  |O
One minor detail. The focal length given on a lens is a value when focusing your camera at infinity. In most cases a so called effective focal length should be used instead, also when calculating Airy disk diameter.

Max
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
One minor detail. The focal length given on a lens is a value when focusing your camera at infinity. In most cases a so called effective focal length should be used instead, also when calculating Airy disk diameter.
I'll be grateful if you explain it in more detail and attach a picture if possible. Effective Focal Length (EFL) is a bit overloaded term. The resulting focal length of multiple lenses is called EFL. Also photographers call EFL a focal length affected by the camera’s crop factor. But looks like you are talking about other things.



Assembled both boards. Everything looks to be ok. I like new buttons much more than previous ones. Magnet ring fits pefrectly, though without magnets, as I haven't received them yet.
 
The following users thanked this post: horstmannhid

Offline Vipitis

  • Frequent Contributor
  • **
  • Posts: 880
  • Country: de
  • aspiring thermal photography enthusiast
The buttons remind me of the Blackmagic Design Micro Cinema Camera.
 

Offline Max Planck

  • Regular Contributor
  • *
  • Posts: 97
  • Country: pl
One minor detail. The focal length given on a lens is a value when focusing your camera at infinity. In most cases a so called effective focal length should be used instead, also when calculating Airy disk diameter.
I'll be grateful if you explain it in more detail and attach a picture if possible. Effective Focal Length (EFL) is a bit overloaded term. The resulting focal length of multiple lenses is called EFL. Also photographers call EFL a focal length affected by the camera’s crop factor. But looks like you are talking about other things.
Please look at the link below.

https://www.optowiki.info/glossary/working-f-number/

 The article is about the effective/working f-number, but basically it is about the same problem, i.e. not focusing your lens at infinity.

If you are calculating the Airy disk diameter as 2.44 x wavelength of light x f-number, you should use the working f-number, not the one given on the lens.

One more thing, stemmer imaging published several tutorials about choosing the correct lens.


Max
« Last Edit: November 26, 2020, 04:51:20 pm by Max Planck »
 
The following users thanked this post: VGN

Offline Bill W

  • Super Contributor
  • ***
  • Posts: 1141
  • Country: gb
    • Fire TICS
Fortunately only a big issue in extreme close up work with long lenses.

An 8mm lens focussed down to 100mm is only reduced in illumination / field of view by 10% as the image is spread out

Bill

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Finally received all components and PCBs!
 
The following users thanked this post: ajw107

Offline rockwell

  • Regular Contributor
  • *
  • Posts: 55
  • Country: de
Can't wait to see the boards populated and heading to first tests.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Can't wait to see the boards populated and heading to first tests.
Me too)



I started from the M-board. As I have made too many changes to the power system since previous hardware revision, I decided to check power supply first, before assembling RAM and FPGA. Generally, new power system works fine!
 
The following users thanked this post: ajw107

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
M-board is fully assembled. No magic smoke, power consumption at expected level, FPGA is alive, both RAM ranks memtest is OK!
 
The following users thanked this post: ddrl46, JanHenrik, therwp, ajw107

Offline bap2703

  • Regular Contributor
  • *
  • Posts: 202
  • Country: io
That looks so professional  :-+
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
That looks so professional  :-+
Thanks! Doing my best)



The P-board is ready!
 
The following users thanked this post: ddrl46, ajw107

Offline dmendesf

  • Frequent Contributor
  • **
  • Posts: 342
  • Country: br
Seems your project is going well. When will you accept beta-testers?  ;D Planning a crowdfunding of some sort?
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Seems your project is going well. When will you accept beta-testers?  ;D Planning a crowdfunding of some sort?
I'm still accepting beta-testers, checkout the very first post of this thread (have updated it for clarity), you will find some details about beta tesing. More information I will provide further.

Planning a crowdfunding of some sort?
Sure, but first I must get sure that every single part of this complex device is working properly. I really don't want anyone to face with hardware problems, even beta-testers.
 

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
Found this "datasheet" for AL1.65X1.65X5D-4F, according to it only 3,8um@90V :-//
https://img.alicdn.com/imgextra/i2/71977092/TB2wz4ybNaK.eBjSZFAXXczFXXa_!!71977092.png

To be sure, I also ordered a set and will test them the same way.
Anyway, I will be very grateful if you share your results with us!

I received the piezo elements today. And unfortunately the "datasheet" you found was correct. Because they were so small, I could not easily use my interferometer meter, but I used a Keyence LK-G37 laser sensor instead. Signal is a bit noiser, but if they would have met their claimed spec, the Keyence should be more than capable also.

Result: with a 0 to 10 volt sine wave, I measure about a +-0.4µm signal, which matches with the 3.7µm@90V from the datasheet...
 
The following users thanked this post: VGN

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Result: with a 0 to 10 volt sine wave, I measure about a +-0.4µm signal, which matches with the 3.7µm@90V from the datasheet...
Thanks so much, _Wim_ , for proving the characteristics! :-+
Yes, this is pity, but ok, at least now we know the truth.

Well, looks like best candidates are still:
Kemet's AE0203D18H18DF from here: https://ru.mouser.com/datasheet/2/212/1/KEM_P0101_AE-1518874.pdf
PI's P-882.51 from here: https://static.pi-usa.us/fileadmin/user_upload/physik_instrumente/files/datasheets/P-882-Datasheet.pdf
Though Kemet's one is avaliable at both Digikey and Mouser, I hope to find something cheaper if it is possible.
 

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
Well, looks like best candidates are still:
Kemet's AE0203D18H18DF from here: https://ru.mouser.com/datasheet/2/212/1/KEM_P0101_AE-1518874.pdf
PI's P-882.51 from here: https://static.pi-usa.us/fileadmin/user_upload/physik_instrumente/files/datasheets/P-882-Datasheet.pdf
Though Kemet's one is avaliable at both Digikey and Mouser, I hope to find something cheaper if it is possible.

Those are indeed expsive, for sure if you need at least 2 per camera.

As you doing your own mechanical design also, would integrating a lever to increase the stroke be an option?
 
The following users thanked this post: JanHenrik, VGN

Offline JanHenrik

  • Contributor
  • Posts: 25
  • Country: de
I really love this project, thus I got two NV3 right away! :3

Since I also like the inbuilt superresolution-stage idea, I redrew the XY-Stage as board in KiCAD and ordered a few. Mine are based on either MLCCs or those shitty AliExpress piezo actuators. However I have very low confidence in MLCCs as they most certainly will crack and the piezos from AliExpress seem to have too little travel. :/
Anyways this will be a fun little thing to play with. If someone wants a board or two, DM me :)

 
The following users thanked this post: _Wim_, VGN

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
Since I also like the inbuilt superresolution-stage idea, I redrew the XY-Stage as board in KiCAD and ordered a few. Mine are based on either MLCCs or those shitty AliExpress piezo actuators. However I have very low confidence in MLCCs as they most certainly will crack and the piezos from AliExpress seem to have too little travel. :/
Anyways this will be a fun little thing to play with. If someone wants a board or two, DM me :)

Never thought about using MLCCCs, but indeed also a piezoelectric material. Will try to measure some this weekend and check if they move any further then the Chinese piezo's...

I like you pcb design!  :-+
 
The following users thanked this post: JanHenrik

Offline JanHenrik

  • Contributor
  • Posts: 25
  • Country: de
Please do, I am interested in the results as well! I have chosen the 1812 package.

You can try to polarize the MLCC to archive a bigger stroke.
Heat to ~150°C -> apply the maximum rated voltage -> keep it in this condition for ~2hrs -> let it cool down -> turn off voltage.

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
JanHenrik, finally! I was waiting for your post! ;)
Looking forward your progress!  :-+



As you doing your own mechanical design also, would integrating a lever to increase the stroke be an option?
Thank you, yes, I though about that. There are special series of APA (amplified piezo actuators): https://www.cedrat-technologies.com/en/products/actuators/amplified-piezo-actuators.html
I'm sure, that these ones are going to be even more expensive... :-\

Anyway, we could use this idea with that cheap 3.7um actuator to increase the displacement level this way, but there some disadvantages:
1. More backlash points (piezo-to-lever, lever-to-stage).
2. We will loose some resulting displacement due to lever flexibility. The stage design is going to be not that simple.
3. Lever based design probably will be quite huge, so we will have to say "good bye" to the reused original aluminium housing, though this is not a big trouble.
 

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
Today I tried with the following 1206 MLCC capacitor (X7R):
(https://www.digikey.com/en/products/detail/samsung-electro-mechanics/CL31B105KCHSNNE/3888405?s=N4IgTCBcDaIIxgOwDYC0SAsjV1QOwBMQBdAXyA)

I was unable to measure any movement with a 20Vpp sine wave (below the noise floor of the Keyence).  I did not try to pre-polarize it, as I could not detect any movement in its original state. Most other mlcc caps I have are 0805 or smaller. I did not have anything larger than this 1206 at hand. I also did not have any  Z5U or Y5V types at hand, these are maybe better for this application?
 
The following users thanked this post: VGN

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
Ok, so I really wanted to know the movement of the MLCC, so I made a setup with the interferometer. But even with the interferometer I could not detect any movement with a 20Vpp 10Hz sine signal.

I connected the setup to the LCR meter and I correctly measure 1µF, so the capacitor is for sure connected.

I even switched the interferometer to FFT mode to see if it would see a 10Hz movement, but nothing. As I do not yet have on optical isolated table, this is really the lowest I can go (if somebody knows a good deal on a small optical table, please let me know).

The 2 peaks in the FFT you can see are the fans of the laptop spinning. Just toughing the mouse already gives a "huge" movement...


 
The following users thanked this post: VGN

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
Just tried 4 Chinese 0805 10µF 10V mlcc caps in series on the interferometer, but even with those no detectable movement. Googling a bit more on the topic, I can find much info about the microphonics related to mechanical bending of an MLCC cap, but no real data about expansion due to applied voltage. I am starting to wonder if they really do this...
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
_Wim_, thanks for your tests! I'm not sure, but probably the voltage level is too low. A man here could achieve 800 nm over 100 V for with 1812 capacitor: https://dberard.com/2015/08/16/mlcc-piezo-actuators/



I want to show some my sketches for those people, who is interested in XY-stage development and piezo actuator testing. I don't think that I will start developing XY-stage right now, as I want to focus on new hardware and prepare it for beta-production. Anyway, hope this will be helpful.

If you look at the M-board schematics, you will find an AUX connector (X4). There are 7-pin of 3V3 IOs, connected to FPGA, single ground pin and two rails of power (3V3 and battery power 4V2-2V5). 7 IOs is enough to support two SPI devices + I2C bus (for reserved purposes).

Also I found a piezo actuator driver HV56020, that integrates a step-up DC-DC converter (up to 225V) and two high voltage amplifiers for driving two piezo actuators.
Datasheet: https://ru.mouser.com/datasheet/2/268/HV56020-Data-Sheet-DS20006335A-1843786.pdf
Mouser link: https://ru.mouser.com/ProductDetail/Microchip-Technology/HV56020T-V-KXX?qs=vmHwEFxEFR%252BS41dx8RulPQ%3D%3D

The only problem with this HV56020 - a poor documentation. I can't understand what kind of transformer is used, no information about it in datasheet. I couldn't find any application notes or examples, evalution boards, etc. Also, no support at the microchip forum: https://www.microchip.com/forums/m1144592.aspx

The schematics are attached:
« Last Edit: December 13, 2020, 02:01:59 am by VGN »
 

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
_Wim_, thanks for your tests! I'm not sure, but probably the voltage level is too low. A man here could achieve 800 nm over 100 V for with 1812 capacitor: https://dberard.com/2015/08/16/mlcc-piezo-actuators/

That could indeed be the case. He only measured 300nm for 100V with a non-prepolarized DUT (which would be only 30nm with 10V in my case), and had a much "larger" capacitor (1812 vs 1206). This would mean the resulting movement in my case is indeed well below the noise floor of my setup, although I would have expected that something would be visible on the FFT...

But in my opinion these very small movements make MLCC caps also not a usable alternative for the purpose of enhancing camera resolution.


Also I found a piezo actuator driver HV56020, that integrates a step-up DC-DC converter (up to 225V) and two high voltage amplifiers for driving two piezo actuators.
Datasheet: https://ru.mouser.com/datasheet/2/268/HV56020-Data-Sheet-DS20006335A-1843786.pdf
Mouser link: https://ru.mouser.com/ProductDetail/Microchip-Technology/HV56020T-V-KXX?qs=vmHwEFxEFR%252BS41dx8RulPQ%3D%3D

The only problem with this HV56020 - a poor documentation. I can't understand what kind of transformer is used, no information about it in datasheet. I couldn't find any application notes or examples, evalution boards, etc. Also, no support at the microchip forum: https://www.microchip.com/forums/m1144592.aspx

That looks indeed like a very interesting chip for this purpose. For ready made boards you could have a look here also:

https://www.apexanalog.com/applications/ink-jet-printer.html

Edit: I did not look carefully enough, these boards still need a high voltage supply which makes them not useful.





 
« Last Edit: December 13, 2020, 07:26:13 am by _Wim_ »
 

Offline _Wim_

  • Super Contributor
  • ***
  • Posts: 1568
  • Country: be
The only problem with this HV56020 - a poor documentation. I can't understand what kind of transformer is used, no information about it in datasheet. I couldn't find any application notes or examples, evalution boards, etc. Also, no support at the microchip forum: https://www.microchip.com/forums/m1144592.aspx

The schematics are attached:

https://www.coilcraft.com/en-us/products/power/coupled-inductors/1-n-shielded-coupled/lpr/za9735/
 
The following users thanked this post: VGN

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
https://www.coilcraft.com/en-us/products/power/coupled-inductors/1-n-shielded-coupled/lpr/za9735/
No way... :palm:  Thanks, _Wim_ . Passive parts designed exclusively for certain ICs are so annoying! BTW, this inductor is not avalible at Mouser/Digikey, though at least we know its parameters. Unfortunatelly there are still a lot of undocumented things, like the feedback resistor divider, diodes, dc-dc step-up caps, Rsht and so on. This is ridiculos to have such a useless datasheet for this quite non-standard application. The only hope is that microchip will finally release any appnote or evalution board. :-\
« Last Edit: December 13, 2020, 12:50:19 pm by VGN »
 
The following users thanked this post: ajw107

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Happy new year everyone! Some updates:

Still working on design port the to new hardware, but finally could get image from sensor with new hardware.

Also the most part of the HDL sources is avaliable on my repo: https://github.com/OVGN/OpenIRV
Vivado is not version control friendly, but many thanks to this man, that designed these special TCL scripts for painless version control of Vivado projects:
https://github.com/barbedo/vivado-git   :-+
Frankly speaking, I was writing code for 20% of time and fighting with Vivado (IDE) the whole residue time. :palm:  |O
But fortunatelly looks like now me and Vivado are understanding each other! ;D

I haven't finished design port yet, so you can find a lot of stubs, no USB, AV, HDMI outptut yet.

I'm using block design graphical feature of Vivado for easier development, but Vivado is quite massive (~45GB), so checkout PDF attachments of complete block design for a quick look.
« Last Edit: January 02, 2021, 01:32:12 pm by VGN »
 
The following users thanked this post: ddrl46, JanHenrik, ArsenioDev, zrq, ajw107

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hello everyone! Finally updates...

I have redesigned the hardware architecture. Previously we had a single DMA and a huge FSM, that was processing the whole data transfer.
Now we have 3 separate modules, each one with it's own DMA:

1. VDMA - universal video outputing core. We have about 6 universal channels to stream video data to multiple consumers. For now only two channels are activated, the LCD and the OSD (on-scree-display for LCD). Remaining channels will be used for HDMI, AV and USB output.
2. DIP (digital image processing). This module includes all image processing cores, i.e. averaging, NUC (non-uniformity correction), BPR (bad pixel replacement), Histogram Equalization + AGC (automatic gain control).
3. Sensor module. This module is used only to control current sensor, i.e. feed it with bias data, commands and grab the video stream.

Why doing this? Because now the hardware design became more scalable. Now there is no need for massive HDL rework to support new sensors, we just have to replace the sensor capturing module with a new one. DIP became more universal too. Yes, it still depends on knowing the sensor active resolution, but I have an idea how to remove this dependency.
 
The following users thanked this post: ddrl46, ajw107

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Also I have developed a special hardware bootloader - OIRVBOOT. This is a special separate "golden" bitstream, that boots first and then switches to the main application bitstream. This hardware bootloader allows to avoid the situations when the device bricks, due to incorrect update or something else. This bootloader for now is quite stupid, I can only stop in bootloder and continue booting the main application. But I'm going to add more features, like firmware update over SD-card, hardware selftesting, selective bitstream boot, backups and so on...

BTW, checkout attached picture with current NOR-flash memory space architecture. There is quite a lot of space to store another bitstream if you would like or keep any other data. The OIRVBOOT will let you to boot any bitstream at the alternate address if needed.

« Last Edit: February 06, 2021, 04:28:45 pm by VGN »
 
The following users thanked this post: ddrl46, ajw107

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
And one more thing, I have started developing the GUI logic. Checkout the attached video:



I decided to use an immediate mode GUI. The GUI that you see is based on Nuklear project: https://github.com/Immediate-Mode-UI/Nuklear
It is pretty simple, pure C, no external dependences, really good. But that was probably not the best choice for my application, as Nuklear do not support custom input, i.e. it is designed to use mouse and keyboard. That perhaps would't be a trouble, as I could map my 4 buttons to some standatd keyboard keys, but Nuklear doesn't allow to use keyboard to navigate  |O, only text input.

So, I'm still looking for a good GUI library with Apache 2.0 or MIT licenses, pure C (no C++), no external dependencies if possible.
I will be glad If you could recommend me something. Yesterday I came across a LVGL library: https://lvgl.io/, github: https://github.com/lvgl/lvgl
LVGL looks promising, worth to try. Any thoughts?
« Last Edit: February 06, 2021, 04:00:22 pm by VGN »
 
The following users thanked this post: ddrl46, rockwell, ajw107, horstmannhid, ericb

Offline ericb

  • Contributor
  • Posts: 16
  • Country: fr
Hello,

First, congratulations for your work, this is simply awesome !!


REMOVED : I didn't see you asked for C gui, not C++

Apologies for the noise.
« Last Edit: February 06, 2021, 08:52:44 pm by ericb »
 

Offline sslupsky

  • Contributor
  • Posts: 16
  • Country: ca
What OS are you using for the firmware?  You might look into the Zephyr project.  It is an embedded RTOS supported by the Linux Foundation.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
What OS are you using for the firmware?  You might look into the Zephyr project.  It is an embedded RTOS supported by the Linux Foundation.
Hi! Oh...forgot to say, I'm using FreeRTOS. It is running on 100MHz 32-bit Microblaze MCU, avaliable RAM size is about 8MB.

First, congratulations for your work, this is simply awesome !!
REMOVED : I didn't see you asked for C gui, not C++
Hi, thanks! Yes, unfortunatelly ImGui, that you have recommended is C++, though there is a cimgui project, that wrappes original ImGui API: https://github.com/cimgui/cimgui
 

Offline pansku

  • Regular Contributor
  • *
  • Posts: 60
  • Country: fi
Looks like a very interesting project and I'm very impressed by the professional level of the design. Bookmarked for sure and seems like I might have to get one of those NV3 modules before summer  :popcorn:
 

Offline swwils

  • Newbie
  • Posts: 3
  • Country: gb
Did you consider helical barrel arrangement for focus instead of the screw? I think a focusing helicoid would be doable with small nylon washers.



This massive one uses actual bearings but you get the idea.
« Last Edit: February 13, 2021, 06:19:23 pm by swwils »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Looks like a very interesting project and I'm very impressed by the professional level of the design. Bookmarked for sure and seems like I might have to get one of those NV3 modules before summer  :popcorn:
Hi, thanks! Consider email me if would like to become a beta-tester (details are in the first post of this thread).



Did you consider helical barrel arrangement for focus instead of the screw? I think a focusing helicoid would be doable with small nylon washers.
This massive one uses actual bearings but you get the idea.
Hi! In fact, no, I haven't considered this way of focusing. I'm not really good with optics, but I think that helical focusing mechanics are mostly needed for zooming, where large lens travel is required. But this camera isn't supposed to have zoom (except digital one). The main idea was to overcome the fixed lens focus distance range limitations (factory focus distance starts from ~3 meters). Fortunately 2mm lens travel screw-based focus is enough to cover distances from ~8cm to infinity. It is also quite cheap and reliable (almost no wear), reused quite a lot of original camera parts. Moreover I have redesigned some parts and fixed some problems with speed, vibrations and noise (will make a post later).

At the same time I understand that somebody probably have much better IR lens. And I don't see any problem to integrate any non-standard IR lens with this camera. Some custom plastic adapter is just needed (FDM printer capabilities should be enough). If you need some more information (dimentions or etc...) to design an adapter, just let me know, probably I could help.
This is the power of opensource.
« Last Edit: February 13, 2021, 07:40:49 pm by VGN »
 

Offline Vipitis

  • Frequent Contributor
  • **
  • Posts: 880
  • Country: de
  • aspiring thermal photography enthusiast
As for lens mount, that is really a difficult one. As I am planning my own thermal camera project, it includes custom housing and a lens mount that supports interchangeable lenses. So no screws or sealing. Just swapping lenses in the field with rear lens caps and mount caps.

I have been thinking about something that locks securely like a PL mount, but the flange distance of what I can work with don't allow that at all. As one of the lens I am planing to use is rather large. It will have to be mounted to rails or a plate. Which makes changing lenses by turning and such more difficult.

I have seen screw mounts in a lot of product, mostly M39 and some smaller stuff among the ThermApp and Thermal Expert dongles. While those sensors are about the same size as the one you are working with, that's not the case for me (I am looking at 21mm diagonal - means I might be able to use a lot of older lenses and get them from sources like Bill). FLIR had their own mount for the higher end cameras like Txxx and Ex5. But it's a mount with electronical connecting for auto focus and saving a lens correction profile.

I am looking forward to what you come up with and might see a way to adapt that for my own plans... Or not.
 

Offline swwils

  • Newbie
  • Posts: 3
  • Country: gb
Ah I understand, great!

I have bought the camera module (AUDI) and am ready to fabricate PCB.

Do you have a current BOM or should I take from schematic?

I plan to make a simple Ge or ZnSe plano-convex lens to use as macro for this system. Like described here:

http://uvirimaging.com/wp-content/uploads/2018/03/Diy-Thermal-Camera-Telephoto-Converter.pdf

This combined with the ready made optics housings from thorlabs (Ø1" Optics) - will make a very very decent and cost effective tool!
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I have been thinking about something that locks securely like a PL mount, but the flange distance of what I can work with don't allow that at all. As one of the lens I am planing to use is rather large. It will have to be mounted to rails or a plate. Which makes changing lenses by turning and such more difficult.
Why not using a special flange adapter to get the proper flange distance for your large lens?

I have bought the camera module (AUDI) and am ready to fabricate PCB.
Do you have a current BOM or should I take from schematic?
I haven't finished testing hardware yet, hope to finish by the end of this month. Also this is supposed to be a kit with fully assembled PCBs + enclosure and other mechanic parts, though for the most brave and experienced of you, I could ship bare PCBs with BOMs, assembling schemes, etc...
 

Offline Spirit532

  • Frequent Contributor
  • **
  • Posts: 490
  • Country: 00
    • My website
Yesterday I came across a LVGL library: https://lvgl.io/, github: https://github.com/lvgl/lvgl
LVGL looks promising, worth to try. Any thoughts?

I highly recommend you go with LVGL. It's extremely simple to implement, can be augmented with 2D acceleration, and is fairly easily extensible.
I'm using it in a commercial project right now, no issues whatsoever.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I highly recommend you go with LVGL. It's extremely simple to implement, can be augmented with 2D acceleration, and is fairly easily extensible.
I'm using it in a commercial project right now, no issues whatsoever.
Hi, Spirit532! Thanks, going to try it!



Meanwhile...updates!

Ok, looks like I have DVI/HDMI working. For now three screen modes are supported: 800x600, 720p and 1080p.
I was a bit surprised with stable operation of the 1080p mode, as it is quite far away from FPGA capabilities according to specifications.
Anyway, it works perfectly over a 3 meter cable:


« Last Edit: February 25, 2021, 11:44:57 pm by VGN »
 
The following users thanked this post: ArsenioDev, RO, Syncronisator

Offline ArsenioDev

  • Frequent Contributor
  • **
  • Posts: 275
  • Country: us
    • DiscountMissiles: my portfolio and landing page
WHOA, that is AMAZING. Cannot wait to add this to my lab debug toolkits as an overhead capture camera, will help A TON, can just dump output into my many HDMI capture inputs.
 
The following users thanked this post: VGN

Offline SilverSolder

  • Super Contributor
  • ***
  • Posts: 6126
  • Country: 00

Truly impressive!

What is a good HDMI capture device?  The ones I have tried in the past all end up over-compressing the image...
 
The following users thanked this post: VGN

Offline smason

  • Newbie
  • Posts: 2
  • Country: us
If youre plugging into a computer, you can look into an Elgato Camlink. Using OBS you could record quite well. this hard worked for me in the past.
 
The following users thanked this post: SilverSolder

Online oPossum

  • Super Contributor
  • ***
  • Posts: 1475
  • Country: us
  • Very dangerous - may attack at any time
From the first post...

Quote
Supported outputs:
1. 320x240 TFT LCD
2. USB UVC (a common webcam protocol)
3. HDMI 480p/720p (probably even 1080p with Spartan-7 FPGA)
 
The following users thanked this post: SilverSolder, ArsenioDev

Offline fest

  • Contributor
  • Posts: 16
  • Country: lv
Received an Audi 4G0980552 part. Although it is missing front window, I was assured by the seller that it's working. By pointing out that without the window, it's worthless for it's intended use, I got it for 160EUR.

Inside is surprisingly intact- no signs of water ingress. I was surprised to find a desiccant bag inside the camera. Judging by the oxidation layer that fell off from the main enclosure screws, nobody had opened it for a very long time, so it appears to have been there by design.

It does appear to be NV3 by the markings on the PCBs.
« Last Edit: March 09, 2021, 08:28:37 pm by fest »
 
The following users thanked this post: VGN

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Received an Audi 4G0980552 part.

Hi, fest! Thanks much for sharing photos!
You have finally approved that Audi's 4G0980552 is exactly NV3.
BTW, internal condition is indeed perfect.
 

Offline fest

  • Contributor
  • Posts: 16
  • Country: lv
Before trying to interface the sensor, I wanted to better understand the signals. I traced required signals on sensor board based on OpenIRV's M-board pinout- see attachments. Unfortunately, the NV3 core board does not have nice testpoints for CMD and BIAS signals, but I found them accessible on resistors on the other side of the board.

VGN, do you know if FPGA queries the sensor at all, before receiving a key?
Just probed the DATA_EVEN and CLK lines- there is a ~75MHz clock and some activity on data line (at least, as much as I could see with DS1052). Judging from the captures in Vue336 interface thread, DATA_EVEN comes from sensor only, so it appears the FPGA is receiving data from sensor even without any security key.
« Last Edit: March 09, 2021, 08:21:09 pm by fest »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
VGN, do you know if FPGA queries the sensor at all, before receiving a key?
Just probed the DATA_EVEN and CLK lines- there is a ~75MHz clock and some activity on data line (at least, as much as I could see with DS1052). Judging from the captures in Vue336 interface thread, DATA_EVEN comes from sensor only, so it appears the FPGA is receiving data from sensor even without any security key.
Exact clock value is 73.636MHz. There is no any encryption or any other secure communication between the camera and FPGA. If you send clock, valid command and valid bias values, you will get valid data on both data lines (even + odd).
 

Offline fest

  • Contributor
  • Posts: 16
  • Country: lv
VGN, you mentioned the difficulty of obtaining bias values for each pixel. Looking at your HDL sources, I now understand that a bias value is sent by FPGA to sensor, for each pixel? That is quite interesting- in that case, it seems to me that there is a significant manufacturing variation for each microbolometer (that is, even more significant than the usual non-uniformity which is handled by the NUC).

Can you share the bias values for your sensor? I wonder if there is any discernible pattern in the values.

Anyway, I'm especially curious about this, because I hooked my unit to a logic analyzer, and the FPGA doesn't seem to send any bias values at all. Consequently after the per-frame command packet, I see something that resembles one image line, and then only the preamble on the data lines from sensor. I wonder if it's a security feature (I've heard about lockdown mode if FPGA receives incorrect data on NV2), or my unit is somehow damaged. There does appear to be some activity on the SPI flash lines, but I'll probably try dumping it next.

Failing that, I wonder what is the way detectors are characterized at the factory. Maybe it's just as simple as as binary search on the bias value for each pixel that yields the most uniform output of the detector.
« Last Edit: March 11, 2021, 11:13:22 pm by fest »
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
VGN, you mentioned the difficulty of obtaining bias values for each pixel. Looking at your HDL sources, I now understand that a bias value is sent by FPGA to sensor, for each pixel?
Yes, that is right. FPGA sends 6-bit bias value for each pixel. (In fact it sends 7-bit, but only 6-bit are significiant).

That is quite interesting- in that case, it seems to me that there is a significant manufacturing variation for each microbolometer (that is, even more significant than the usual non-uniformity which is handled by the NUC).

This is also true. There are two ways to solve this problem. Bias data and command word can be retrieved from original board NOR flash (I have some success here). Another way - to recalculate this values over a two-point calibration.

Can you share the bias values for your sensor? I wonder if there is any discernible pattern in the values.
Follow this reply: https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/msg3254678/#msg3254678
I used fixed 0x25-0x26 bias value for all pixels.

Anyway, I'm especially curious about this, because I hooked my unit to a logic analyzer, and the FPGA doesn't seem to send any bias values at all. Consequently after the per-frame command packet, I see something that resembles one image line, and then only the preamble on the data lines from sensor. I wonder if it's a security feature (I've heard about lockdown mode if FPGA receives incorrect data on NV2), or my unit is somehow damaged. There does appear to be some activity on the SPI flash lines, but I'll probably try dumping it next.
This is ok, camera do not output any bias values at this point, it waits for some command from the control unit. Don't worry, your unit is not damaged, as you see preambles at the data lines (10101010101010). Preambles are generated by sensor, that means it is alive. As I said previously, there is no any secure protocols between sensor and FPGA. First three lines - are a pipeline garbage, it doesn't look like to be a thermal data, no correlation with scence or sensor temperature. The first image line you will get at the 4-th line output. But now you see zeros after preambles 10101010101010, as you do not send any bias at all. Also there is a huge pipeline delay (3 lines) between transfer of the bias values and pixel output. After command transfer we start sending bias values, then we will see 3 strange data lines from sensor (I call it pipeline garbage), only after that we will get valid thermal data. Checkout ISC0901_capture.v design file.

Failing that, I wonder what is the way detectors are characterized at the factory. Maybe it's just as simple as as binary search on the bias value for each pixel that yields the most uniform output of the detector.
I think you are right. There is only 64 levels of bias. We just have to take 128 pictures from two different simple blackdoby sources, find the bias values for each pixel that gives the best pixel dynamic output range. But picture will not be uniform at this step. We will have to calculate individual pixel gain value to get uniform image.
In fact I have already tried to calculate pixel gain values for fixed 0x25 bias. Quality of all videos and photos from my thermal camera in this thread is based on my own custom calibration.
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
Hello, some updates...

Finally have USB UVC working. Tested on linux, windows and even android. Sorry, no video for this time, but a photo.
I haven't pushed new commits for a long time, going to fix that in a few days. By the end of this week I'm going to finish AV (analog video), this is the last hardware subsystem, that I must check.

Also I have started working at documentation about the sensor, including pinout, waveforms, command pattern (at least the things I know and understand right now). This document will be updated in case of any new details.

 

Offline bap2703

  • Regular Contributor
  • *
  • Posts: 202
  • Country: io
I think you're doing too well by yourself for people to post useful comments.
So in the end I can just say you're doing an amazing work  :-+
 
The following users thanked this post: SilverSolder, VGN

Offline Hydron

  • Super Contributor
  • ***
  • Posts: 1099
  • Country: gb
I agree that there is some fantastic work going on. I only wish I had more time available to look into your design in more detail - I would like to have a go at interfacing a ULIS sensor to it in place of the FLIR one, as I have a couple without working cores attached (both VGA and QVGA).
 
The following users thanked this post: VGN

Offline fest

  • Contributor
  • Posts: 16
  • Country: lv
Which ones do you have? Most of them seem to have analog output, although some of the newer ones have digital output (which, although is simpler protocol wise, is not compatible with ISC0901B0).
 

Offline VGNTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: am
I think you're doing too well by yourself for people to post useful comments.
So in the end I can just say you're doing an amazing work  :-+
Thanks for your support! It really helps to keep working)

I agree that there is some fantastic work going on. I only wish I had more time available to look into your design in more detail - I would like to have a go at interfacing a ULIS sensor to it in place of the FLIR one, as I have a couple without working cores attached (both VGA and QVGA).
Thanks! I think this is possible to do, but you will have to make a special adapter board for your sensors, check and tune power rails to suit ULIS power requirements and implement a HDL design for FPGA that will capture raw data and feed to the image processing pipelines. I keep in mind, while developing, that image processing cores should be able to support other sensors and resolutions.

Which ones do you have? Most of them seem to have analog output, although some of the newer ones have digital output (which, although is simpler protocol wise, is not compatible with ISC0901B0).
For this case we have 20 high performance IOs at the sensor connector of the M-board. I think this is enough to connect ADC, which can be placed on the special adapter board. Yep, this is quite tricky, but worth doing if you want to bring alive some really cool and expensive sensor.
 

Offline Hydron

  • Super Contributor
  • ***
  • Posts: 1099
  • Country: gb