Author Topic: OpenIRV. ISC0901B0 (Autoliv NV3, FLIR E4/5/6/8) based opensource thermal camera  (Read 3749 times)

0 Members and 1 Guest are viewing this topic.

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Hello everyone!

This topic is about my attempt to develop an opensource high performance and multifunctional thermal camera based on a ISC0901B0 sensor.

I use thermal imaging to diagnostic malfunctions of different devices under development or repair. Five years ago I have developed a small camera based on FLIR Lepton 2 80x60. It works pretty well, but in fact this is a toy for me, and I was looking for something better. Unfortunately FLIR EX series didn't suit for me too for many reasons, including poor fixed focus lens, a bit inconvenient form factor, price, etc...
I'm also a supporter of opensource projects, but all of them are still based on FLIR Lepton 2/3 products or some other tiny sensors.

Once an Avtoliv NV3 camera module occured in my hands. I've disassembled it and found out that this camera module has a great potential and perfectly suits my new project, because:

1. It is based on very popular ISC0901B0 336x256 sensor. Popular, because NV3 camera is used on tons of different vehicles till now, you can find this spare part very easily. FLIR EX and ETS320 are based on the same sensor.
2. It is an external shutter based camera. External shutter allows to significially improve the image quality.
3. It has a pretty fast (15mm in dia) telescopic lens! This lens is much better that FLIR EX lens.
4. Pretty nice form factor (60x62mm)

I decided to develop a camera that will be multifuntional and can be used in many different ways and applications:
1. Hand tool, like FLIR EX
2. Stand tool, like ETS320
3. Thermal camera for robotics and UAVs, like FLIR Vue

Here is how it looks like now:
[attach=1][attach=2][attach=3][attach=4][attach=5][attach=6]

Supported outputs:
1. 320x240 TFT LCD (40-45 fps)
2. USB UVC (a common webcam protocol)
3. HDMI 480p/720p
4. AV (NTSC/PAL and many other formats if necessary)
5. Micro SD card (SPI mode + SDIO x4 in future)
6. 3-pin configurable 5V tolerant GPIO to support any low speed interface (UART, SPI, I2C, 1-Wire, PWM, PPM, SBUS...)


Power:
1. Standard Li-ion 18350 battery. Battery life is about 3-3.5 hours (with 1100ma battery), depends on number of active outputs, LCD backlight, etc.
(I'm still working on increasing this value, but I think that reaching 4 hours is a good result.)
2. USB charge (<500ma). USB power is enough for continues camera run with all outputs enabled.
3. External power over aux connector 5.0-35.0V (i.e. up to 8S battery) for UAV mode.


Some features:
1. Adjustable motorized focus system (details later).
2. Supports frame rate up to 60fps (details later).
3. Two standard 1/4" metal mounts on the top and bottom.

More photos/videos and detailed stories of each side of this device will be later too.

P.S. Project sources are not available right now, but I have plans to publish them later (the repo needs some review and cleanup before publishment).
« Last Edit: July 12, 2020, 01:50:57 am by VGN »
 

Offline Cat

  • Regular Contributor
  • *
  • Posts: 73
  • Country: de
Awesome, this looks very promising :-+
I'm looking forward to learn more about your project, especially how you interfaced the µbolometer and implemented UVC.
Didn't think it would be possible to get a motorized focus inside the tiny housing.
Keep up the good work!
On the Internet, nobody knows you're a cat.
 
The following users thanked this post: VGN

Online Fraser

  • Super Contributor
  • ***
  • Posts: 9543
  • Country: gb
What a superb project, well executed and very professional in its appearance   :clap:  :-+

As I own one of these cameras I will await further details with GREAT interest  :-+

Fraser
 
The following users thanked this post: VGN

Offline zrq

  • Contributor
  • Posts: 25
  • Country: cn
Quite interested in the interface with the sensor and the reverse engineering. Can't wait to see the code! (and hopefully writeup)
 
The following users thanked this post: VGN

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Awesome, this looks very promising :-+
I'm looking forward to learn more about your project, especially how you interfaced the µbolometer and implemented UVC.
Didn't think it would be possible to get a motorized focus inside the tiny housing.
Keep up the good work!
Thank you! Looking ahead, I used FX2LP for USB. FPGA is streaming data over 8-bit bus. As a start point I used this Cypress project: https://community.cypress.com/docs/DOC-14406. More details further... With regard to the focus system, first I didn't believe too)

What a superb project, well executed and very professional in its appearance   :clap:  :-+
As I own one of these cameras I will await further details with GREAT interest  :-+
Wow, many thanks! :)  As you are owner of one of these cameras, I hope for your help with testing ;)

Quite interested in the interface with the sensor and the reverse engineering. Can't wait to see the code! (and hopefully writeup)
Hi! The interface with the sensor is, obviously, the most tricky moment. In fact, there are a lot of things that I still don't understand, but I hope for community help.
 

Offline conmega

  • Contributor
  • Posts: 22
  • Country: us
I also have an NV3 and was working on reverse engineering efforts to interface with the original circuitry.
I just got some boards I made for interfacing to the NV2 going out to some friends to help with development/testing. Although its mostly used as a test platform not such a professional looking project as yours. I look forward to your progress and wish you luck!
 

Offline tmbinc

  • Regular Contributor
  • *
  • Posts: 215
Very cool project!

Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?

Can't wait to see more details!
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
I also have an NV3 and was working on reverse engineering efforts to interface with the original circuitry.
I just got some boards I made for interfacing to the NV2 going out to some friends to help with development/testing. Although its mostly used as a test platform not such a professional looking project as yours. I look forward to your progress and wish you luck!
Hi! I think we could colaborate for better result. I don't have NV2, as I know, this is a previous generation camera, it has an internal shutter (behind the lens). Sure you know, there is a great work about NV2: https://debugmo.de/2018/12/autoliv-nv2-teardown/

Very cool project!
Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?
Can't wait to see more details!
Hello! Thanks much! Yes, you are right. I refused using original circuitry in my project, because it may not survive, when the front window is broken. Also original electronics limits some opportunities.
« Last Edit: July 13, 2020, 10:44:12 am by VGN »
 

Offline Max Planck

  • Regular Contributor
  • *
  • Posts: 79
  • Country: pl
Bought some time ago an NV3, but never had time to work on it.

Thus, I'm really impressed by the project.
Are you using just the FPA or also some parts of the NV3?

Max
 
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Do I understand correctly that you're using the sensor and other hardware, but not the PCBs/FPGA of the NV3?
Thus, I'm really impressed by the project.
Are you using just the FPA or also some parts of the NV3?
Thanks you! Ok, I thinks this will be the start point of the story.

I have three cameras, two of which have cracked front windows (btw cost me under 100$ for each). I didn't expect that condition of the most critical parts, i.e. sensor and lens will be perfect. In fact all three sensors are alive with amount of bad pixes <1%.

[attach=7]

As you know, the front window is very fragile. As soon as it catches a stone from the road and cracks, your camera will soon stop working, because of water. Some water damage pictures:
[attach=1][attach=2] [attach=3] [attach=4][attach=5]

Nevertheless the sensor itself in located in the center of the camera module, and incomming water do not reach it, as the water level do not rise above the bottom edge of the broken window. Perhaps that's why both sensors of drowned cameras are working. Of course you should be carefull, there is no 100% garanthy that sensor will work, but at the same time the probability is quite high. The shutter motor can also be easily replaced if necessary.

You can also see a lot of small dents at the surface of the windows. But the surface of all lens are in perfect condition. I think this is because lens is located very deep inside of the enclosure and also protected by broken window. Anyway that gives a chance to bring alive very poor cameras.


Here you can see parts that I have reused from the original camera:
[attach=6]

The front window and cap is optional. I don't really use them, because it is an additional obstacle for IR radiation and the device itself is not water or dust protected. It is designed mostly for home and limited outdoor usage.
« Last Edit: July 14, 2020, 06:27:23 pm by VGN »
 
The following users thanked this post: zrq

Online Fraser

  • Super Contributor
  • ***
  • Posts: 9543
  • Country: gb
So if it is just the casing, lens, FFC flag and Microbolomter PCB that is used in your design, presumably this project could be adapted to the earlier NV2 model ?

You have basically built a thermal imaging camera from scratch which is a significant achievement when you do not have the datasheet for the microbolometer or its recommended bias voltage values. Building the back end video processing package and firmware is a further very impressive achievement. You have skills  :-+ Much respect for you  :clap:

Fraser
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
So if it is just the casing, lens, FFC flag and Microbolomter PCB that is used in your design, presumably this project could be adapted to the earlier NV2 model ?
Yes, you are right. I believe it can be adapted to any camera, that needs deep raw image processing.

You have basically built a thermal imaging camera from scratch which is a significant achievement when you do not have the datasheet for the microbolometer or its recommended bias voltage values. Building the back end video processing package and firmware is a further very impressive achievement. You have skills  :-+ Much respect for you  :clap:
Thanks much! Bias values were picked by hand, I'm still looking for a way to calculate this values properly. Do you have any knowleges about that? After tons of experiments I found out that this is 6-bit value per pixel and could properly feed them to FPA core.
 

Online Fraser

  • Super Contributor
  • ***
  • Posts: 9543
  • Country: gb
The bias values are normally provided by the fabricator of the microbolometer and are unique to each one. The user of the microbolometer then uses these values as a start point from which they are fine tuned for the cameras particular application, temperature range etc. The bias voltages can be fine tuned for best NeTD. Sadly I do not have any bias voltage values for FLIR microbolometers.

Fraser
 

Offline Treehouseman

  • Supporter
  • ****
  • Posts: 56
I've been thinking that it could be possible to extract the bias values from the flash chips, I would imagine it's just a semi standardized mapping that if you compare 2 camera's flash images and see what the differences is it may stand out. It would then be a task to see how they're storing it but with the NV2 given that we have the code from it's softcore, the softcore may load in the values, and that could tell us how they're stored. Especially if they keep the same structure they used for debugging where you could also send it values manually for initial calibration.

While we don't currently have the program from the nv3 it's possible it uses the same data encoding. On top of that we could also see how it's used in a Flir E4 or a Flir E40 (NV3 and NV2 respectively) and see if it's the same thing. I would assume they just drop the same table of values in.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Well, there are a lot of things to tell about, so for simplicity I'm going to move step by step from outside to inside.

The whole enclosure consists of four parts:
1. Top (where the display is mounted)
2. Middle
3. Bottom
4. Original black casing

All parts were developed for CNC machining, but I decided to try SLA 3D printing for prototype. Of course, the quality is not super, there are some wracks at the surface and edges. Anyway I'm sure CNC machining with further casting plastic in silicone molds will give much better quality.

[attach=8]

The bottom:

[attach=1]

The middle:

[attach=2]

The metal mounts is nothing but a 1/4" to 3/8" thread adapter. Fitting the adapter from the inside prevents possiblity of unwanted unscrew:

[attach=3][attach=4][attach=5]

The top:

[attach=6]

Finally I had to increase the lens hole a little to prevent field of view narrowing, because previously I had to raise this black case up to place the focus mechanics.
There is a round boarder that helps to make a neat cutout. This cutout will still allow you to fit the front window if you need it, but maybe you will also have to increase the hole of the window cap too.

[attach=7]
« Last Edit: July 14, 2020, 12:17:32 am by VGN »
 
The following users thanked this post: Fraser, Max Planck

Online Fraser

  • Super Contributor
  • ***
  • Posts: 9543
  • Country: gb
Great to read this thread as it gradually reveals the work you have done. Loving this  :-+

Fraser
 

Offline Max Planck

  • Regular Contributor
  • *
  • Posts: 79
  • Country: pl
First technical question.
I know, it is maybe to early at this stage, but have you taken into account the heating of your camera enclosure during operation (electronics, battery, external heat sources) and its temperature changes on camera (FPA) operation?

Max
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Great to read this thread as it gradually reveals the work you have done. Loving this  :-+
Thanks a lot!

First technical question.
I know, it is maybe to early at this stage, but have you taken into account the heating of your camera enclosure during operation (electronics, battery, external heat sources) and its temperature changes on camera (FPA) operation?
Yes, it's taken into account. As you know, all FPA pixel are different, they all have different response to the same amount of IR radiation. So to solve this problem and make the image uniform we should determine a gain and an offset parameter for each pixel. Gain parameter changes with the camera temperature quite slowly, whereas the pixel offset changes at a temperature difference of ~0.2°C. To compensate this temperature drift FFC (Flat-field correction) procedure is initiated. We use external shutter to close the front of the lens and assuming that shutter surface is uniform, we perform recalculation of offset parameter, improving the image. You can run FFC manually any time you want to improve the image quality or enable automatic mode, when firmware monitors enclosure temperature by integrated temperature ICs and initiates FFC periodically.
« Last Edit: July 14, 2020, 04:17:05 pm by VGN »
 
The following users thanked this post: Max Planck

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
The display module.

[attach=1][attach=2][attach=4][attach=5]

This is the biggest LCD that I could find to fit in 60x62mm dimentions. Also manufacturer promises long term supply for this part (10 years since 2018).

Brass thread inserts are glued into plastic by cyanoacrylate. That looks ugly, that's why plastic molding is preferable for manufacturing.

Buttons cover is made of platinum cured silicone with black pigment. It's pretty reliable and looks good. As you see the button's plunger is quite small, so I decided to place additional 0.3mm plastic plunger that will allow to increase button's press area. Plastic plungers were cutted by hand and looks also ugly, but laser cut will fix that.

There is also a small bug, the buttons board should have additional mounting hole in the center. Will be fixed in future revision.

Few words about electronics. The CLC filter reduces power noise from AUX port. DC-DC converts 5.0V-35.0V to 4.5V. Unfortunately, there was no enough space to fit this regulator on the mother board, that's why it is here. All inductors are shielded to reduce possible EMI noise. LCD backlight is powered by constant current regulation IC (not PWM) also to minimize the power noise level.

[attach=3]



Feel free to ask questions and criticize the design, that may help to make it better.
« Last Edit: July 14, 2020, 07:38:00 pm by VGN »
 

Offline newex

  • Contributor
  • Posts: 17
  • Country: 00
The amount of work you have done is amazing.
I think it makes no sense to criticize the design, because we do not see the schematic diagram, but it looks great and professional.
 

Offline Vipitis

  • Frequent Contributor
  • **
  • Posts: 675
  • Country: de
  • aspiring thermal photography enthusiast
I am more than impressed by the effort so far. Wish I had the knowledge, experience and tools to mount such a project. But really shows what can be done... and motivates me to move on with my project (in 14 days.)
 

Online Fraser

  • Super Contributor
  • ***
  • Posts: 9543
  • Country: gb
I am loving this thread. I very much look forward to seeing future updates. Your camera project looks very professional and not unlike what I have seen in R&D prototyping. Very professional in all respects.

I might add that even with the skills to create the mechanical and electronic design, that is not enough for such a project, as I well know. The fact that you also have the skills required to write the code for the hardware is very impressive :-+

Fraser
« Last Edit: July 14, 2020, 11:01:08 pm by Fraser »
 

Offline Max Planck

  • Regular Contributor
  • *
  • Posts: 79
  • Country: pl
Yes, it's taken into account. As you know, all FPA pixel are different, they all have different response to the same amount of IR radiation. So to solve this problem and make the image uniform we should determine a gain and an offset parameter for each pixel. Gain parameter changes with the camera temperature quite slowly, whereas the pixel offset changes at a temperature difference of ~0.2°C. To compensate this temperature drift FFC (Flat-field correction) procedure is initiated. We use external shutter to close the front of the lens and assuming that shutter surface is uniform, we perform recalculation of offset parameter, improving the image. You can run FFC manually any time you want to improve the image quality or enable automatic mode, when firmware monitors enclosure temperature by integrated temperature ICs and initiates FFC periodically.
Yes, you have two things, NUC and thermal dirft caused by several factors. I was asking about the latter. You are correcting it using a shutter. My question was about the dynamics of that process, i.e. how often the shutter has to be activated to keep a decent image quality/temperature measurement accuracy.

Knowing about some of the challenges you were and are facing, I'm impressed.

Max 
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
The amount of work you have done is amazing.
I think it makes no sense to criticize the design, because we do not see the schematic diagram, but it looks great and professional.
Thank you! Well, yes, I realize that this hard to criticize the design without the schematics, but I mean, maybe you have some global ideas that do not require schemes currently. The problem is that I'm going to make quite a lot of changes for the next hardware revision and I think that it would be better if I publish the very last schematics. I believe we will have enough time to discuss and make changes to design if necessary before I send new files for manufacturing.

I am more than impressed by the effort so far. Wish I had the knowledge, experience and tools to mount such a project. But really shows what can be done... and motivates me to move on with my project (in 14 days.)
Thanks much! I wish you good luck with your project!

I am loving this thread.
...
Many thanks, Fraser!

Yes, you have two things, NUC and thermal dirft caused by several factors. I was asking about the latter. You are correcting it using a shutter. My question was about the dynamics of that process, i.e. how often the shutter has to be activated to keep a decent image quality/temperature measurement accuracy.
Knowing about some of the challenges you were and are facing, I'm impressed.
Thank you, Max! Basically one should perform FFC every time when camera's temperature changes for a some threshold value up or down. Also there is no need to perform a FFC if temperature is stable, the image quality will not get worse in time. I can't exactly say how often the shutter has to be activated, as it depends on the environment temperature, how fast the camera heats up and the temperature change threshold value. I don't think that this a point of worry, until you use it on UAV, because FFC causes an image freeze for a short period of time (~1 sec). There are two ways to mitigate this problem. First - you can initiate FFC yourself, second - the firmware can display a countdown before FFC to help the pilot get ready.
« Last Edit: July 15, 2020, 05:20:00 pm by VGN »
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Peripheral board (P-board):

[attach=1][attach=6]

Some descriptions:

[attach=2][attach=3]

This board do not process image at all, but provides output path to different interfaces, manages battery power, powerpath logic and some other stuff. D-board (display) and B-board (buttons) are connected to it through 12-pin and 5-pin JST connectors. You can also see a 6-pin JTAG header that is used for FPGA debug on the M-board (main), which I will show futher.

[attach=5][attach=4]

Some device safety features:
  • All IOs at HDMI, USB and 7-pin AUX port are ESD protected.
  • USB 5V power line has overcurrent (>500ma) and overvoltage (up to 20V) protection.
  • AV output has an output buffer with short-to-ground and short-to-battery (up to 18V) protection.
  • AUX external power line has high side reverse polarity protection. (high side protection type is probably not the best choice...)
  • Three GPIOs at the 7-pin AUX port that goes directly to FPGA are protected by 5V tolerant bidirectional buffers.
  • Battery reverse polarity protection. Magic smoke will not appear even if you install battery wrong for a continues period of time.
  • Battery is also protected by a special charge/discharge voltage/current protection IC like in factory protected Li-ion batteries. I refused using protected Li-ions, as their length can vary too much from manufacturer to manufacturer, you simply won't be able to insert the battery into holder, meanwhile dimentions of unprotected batteries are much more predictable.

Other features:

  • RTC provides timestamps for SD card filesystem.
  • Temperature sensor.
  • Power reset button.
« Last Edit: July 16, 2020, 12:12:50 am by VGN »
 
The following users thanked this post: joe-c

Offline joe-c

  • Frequent Contributor
  • **
  • Posts: 329
  • Country: de
    • Joe-c.de
its a really awesome project.
I am very interested.

currently, I work on an open-source solution too, but its more an interface of an existing Camera, not a completely new device.

I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.

anyway... really nice. Thanks for show it.
best regards
FLIR E4 hardware mods 
Freeware Thermal Analysis Software: ThermoVision_Joe-C
Freeware Remote Software for some FLIR Cameras: Ex Tools
 

Offline Ismsanmar

  • Contributor
  • Posts: 21
  • Country: es
VGN, incredible work. Another one here interested in the possibility of adapt your solution for the NV2 ISC0601 sensor. I'm another one with one of those lying around.


I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.

The problem with that camera is that only goes at 15fps on line detection mode
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
its a really awesome project.
currently, I work on an open-source solution too, but its more an interface of an existing Camera, not a completely new device.
I see not a visual camera... right?
maybe it can be combined with this one: https://openmv.io/products/openmv-cam-h7
and let the openmv board deliver a corner detection overlay for a "like MSX" image.
Thank you!
No there is no visual camera, but I thought about that. You know, I held Flir EX in my hands once. And I can't say that MSX technology impressed me a lot. MSX cannot help when FPA's resolution is low itself or when you cannot change the lens focus. Also image alignment is not possible at short distances. Looks like MSX brings more engineering headache than benefits. Are there a lot of people using MSX? Maybe you have some arguments that can change my mind?

VGN, incredible work. Another one here interested in the possibility of adapt your solution for the NV2 ISC0601 sensor. I'm another one with one of those lying around.
Thanks!
Yes, I think it's possible to adapt this camera for NV2 sensor. But I don't have it, and know completely nothig about ISC0601 sensor. If you have this camera, you could investigate the pinout, how is it powered, how many digital IOs should be used to connect it to FPGA, also IO voltage. In future hardware revision I could make some changes for main board, that will help NV2 owners to bring alive this sensor too.
 

Online Hydron

  • Frequent Contributor
  • **
  • Posts: 473
  • Country: gb
I agree that a visual camera in the MSX style (i.e. for edge enhancement etc) isn't necessary with a high resolution (>240p) thermal imager. If you're using the camera for security or something however it can be useful as a separate imager, e.g. to zoom in on someone that you've detected with a wider angle thermal imager.

I'm very impressed with this project - I know a little (mostly theoretical) about thermal imager design and it's a significant undertaking, normally involving a commercial team and significant money! Very interested to learn more.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
I'm very impressed with this project - I know a little (mostly theoretical) about thermal imager design and it's a significant undertaking, normally involving a commercial team and significant money! Very interested to learn more.
Thanks!

I agree that a visual camera in the MSX style (i.e. for edge enhancement etc) isn't necessary with a high resolution (>240p) thermal imager. If you're using the camera for security or something however it can be useful as a separate imager, e.g. to zoom in on someone that you've detected with a wider angle thermal imager.
Integration of visual camera is quite challenging, though not impossible. The problem is that right now there is no free IOs on headers that connect boards with each other, while IO muxing is also not easy. I also doubt that, for example, in UAV mode this visual camera will be better that specialized FPV one. Though at the same time you could easily multiplex outputs of FPV visual and thermal camera. Generally, I think there should be really strong arguments for visual camera integration.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Main board (M-board):

[attach=1] [attach=4]

The brain of the whole device is a Spartan-6 FPGA. As a memory buffer I decided to use HyperRAM. HyperRam memory controller was also developed from the scratch, as there was no high performance implementation of this IP-core in the opensource space. In fact this IP-core is my separate project.
You could ask me, why didn't I use free FPGA's integrated memory controller with DDR2/DDR3 memory. The thing is a very high power consumption, DDR3 + integrated controller would burn even much power, than the whole camera consums itself.

The red bond wire is a bug fix. Though is not the last bug. Using Spartan-6 was a mistake. This FPGA is quite powerful, but also old, has a very poor support, no updates, some bugs in IDE, endless loops in peripherial drivers... :palm:
That's why I'm planning to replace it with Spartan-7 XC7S50 in the same package. I hope the migration will be easy, as previously I had some experience with Xilinx 7-series FPGAs.

The whole stack:
[attach=2] [attach=3]
« Last Edit: July 17, 2020, 01:49:39 pm by VGN »
 

Online Hydron

  • Frequent Contributor
  • **
  • Posts: 473
  • Country: gb
The brain of the whole device is a Spartan-6 FPGA. As a memory buffer I decided to use HyperRAM. HyperRam memory controller was also developed from the scratch, as there was no high performance implementation of this IP-core in the opensource space. In fact this IP-core is my separate project.
You could ask me, why didn't I use free FPGA's integrated memory controller with DDR2/DDR3 memory. The thing is a very high power consumption, DDR3 + integrated controller would burn even much power, than the whole camera consums itself.
Commercial camera cores use things like PSRAM (psuedo-static RAM) for this reason also - lower power consumption than DDR2/3 plus controller.
 

Offline conmega

  • Contributor
  • Posts: 22
  • Country: us
The red bond wire is a bug fix. Though is not the last bug. Using Spartan-6 was a mistake. This FPGA is quite powerful, but also old, has a very poor support, no updates, some bugs in IDE, endless loops in peripherial drivers... :palm:
That's why I'm planning to replace it with Spartan-7 XC7S50 in the same package. I hope the migration will be easy, as previously I had some experience with Xilinx 7-series FPGAs.

It should be pretty simple yes... Stuff like clocking/serdes units have minor changes and such but I have done a bit of similar porting in the past. Even from Altera parts to Xilinx parts from tmbinc's Altera based FPDLink receiver over to a Xilinx 7 series part. It was only a few days of banging my head against the wall. Mostly because I had to learn migen :) Otherwise just some primitives that should need to be changed and some logic wrapped around possibly.
 

Offline ArsenioDev

  • Regular Contributor
  • *
  • Posts: 56
  • Country: us
Holy crap this is basically the HOLY GRAIL for my projects. I'll be watching this with INTENSE interest, it's essentially my dreams come true.
 

Offline Miek

  • Contributor
  • Posts: 26
  • Country: gb
Since you're planning an FPGA swap anyway, it's worth considering the Lattice ECP5 series - the open-source toolchain for them is very good these days. Also, it would probably tempt more people to get involved once you open source the project.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Holy crap this is basically the HOLY GRAIL for my projects. I'll be watching this with INTENSE interest, it's essentially my dreams come true.
Thanks! All the fun is ahead  ;)

Since you're planning an FPGA swap anyway, it's worth considering the Lattice ECP5 series - the open-source toolchain for them is very good these days.
I was thinking about that long ago, when realized that swap is needed. Well, yes, open-source toolchain is very cool and important (especially for security based projects), but I'm not completely sure, that it is worth doing that. I have several arguments:

1. I've already started migration to S7. I'm also afraid that migration from Xilinx to Lattice will be not easy, nor fast. Whole design is AXI4/AXI4-Stream based. I use Xilinx DMA, SERDES... Generally, Lattice swap will cause too massive rework.

2. Lattice ECP5 40nm is a pretty good power optimized solution, but power consumption is not a problem even with Spartan-6 45nm. I hope Spartan-7 28nm will burn even fewer amount of power.

3. I have almost no experience with Lattice, so I'm a bit confused with ECP5 IO performance. ECP5 has 4 SERDES channels, while Xilinx has SERDES on each IO. Will it be enough for HyperRAM and HDMI simultaneous work? BTW, not also sure that ECP5 supports TMDS standard (for HDMI). Is it possible to stream HDMI directly from FPGA without a PHY? If yes, why do they use here HDMI PHYs with ECP5 for such a simple project?
http://www.latticesemi.com/-/media/LatticeSemi/Documents/UserManuals/1D2/FPGA-UG-02036-A.ashx?document_id=52240

Meanwhile Spartan-7 with 1080p 60Hz and even 1440p 30Hz output:
http://labs.domipheus.com/blog/hdmi-over-pmod-using-the-arty-spartan-7-fpga-board/

4. If we try to compare capacity somehow: LFE5UM-45 < XC7S50 < LFE5UM-85
LFE5UM-45 should be excluded, as it looks like to be equal to XC6SLX45, that I use now, but keep in mind that ECP5 is LUT4, while S6 and S7 are LUT6.
Obviosly, we should choose between XC7S50 and LFE5UM-85.

Digikey prices:
LFE5UM-85F-7BG381C ~38$
XC7S50-1CSGA324C    ~53$

Lattice price is lower, but not as much, also FPGA's price is not a big part of the total device cost.

Сorrect me if I'm wrong.


Also, it would probably tempt more people to get involved once you open source the project.
It will be open source anyway, please, be patient  :)
Let's resolve hardware issues first. I hope next week I will commit preliminary schematics (rev.C) for community review and discussion.
« Last Edit: July 21, 2020, 01:23:49 am by VGN »
 

Offline firehopper

  • Frequent Contributor
  • **
  • Posts: 386
  • Country: us
also would be nice for those uninitiated, what to look for to get these modules. the nv3 ones.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
also would be nice for those uninitiated, what to look for to get these modules. the nv3 ones.
66549322653 or 9322653 - BMW
A2229052805 - Mercedes
4G0980552/4G0980552A - AUDI/WV?
 
The following users thanked this post: Fraser, BravoV, Hydron, ArsenioDev, ajw107

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
The focus system:

[attach=11]

There are four SLA printed parts:

[attach=1][attach=4][attach=5][attach=6][attach=7]

1. wheel
2. worm
3. motor mount
4. worm mount

Focus board:

[attach=2][attach=3]

One of the most tricky things was the connection of focus wheel with the lens holding ring. You cannot imagine how much I hate glue, that's why you didn't see any glue connection in the whole design. But...unfortunatelly there were no other cheap and easy way to connect this two parts. Small drops of cyanoacrylate on each tip of the focus wheel allows to reliably connect this two parts so, that I cannot break it by hand. I also tried epoxy, but the adhesion was very poor, so cyanoacrylate is the best for gluing this type of plastic. At the same time the whole focus mechanism is quite lightweight and there is no need to apply much force to make the lens move.

[attach=9][attach=10][attach=8]

The lens holding ring thread allow to make 5 turns of the focus wheel, until it fully unscrews and falls out, that's why we can use 4 turns to adjust the focus. The focus distance changes non-linearly with the wheel position, and that is very good, because a single turn from the fully closed position allows to cover focus distances from ~30cm to infinity, while for macro you will need to make few other turns.

You can also see a photomicrosensor that allows to precisely control the lens position. It is mostly needed to keep the lens away from endpoints, preventing lens lock and unscrew. Also this sensor will be helpful for autofocus! The wheel has 25 teeth, the worm has a single thread and 4 blades, that cross the photomicrosensor slot, so if we detect both rising and falling edges of the pulse, we will have 25x4x2 = 200 points of lens position per a single turn, i.e. accuracy of 1,8 degree.

The focus board thickness is 3mm, it also has 2 additional support points (two 10mm white threaded spacers), that allows to keep sufficient construction rigidity.
The worm mount has a special brass sleeve, that helps to minimize the wear.

The focus motor is a 6mm dia coreless motor with 3 stage built in planetary gearbox (1:136). It is very powerful and consums few energy, but motor vibration is high and rigid connection to the PCB causes some sound amplification. But I have a solution for this, the motor mount is to be redesigned, I'm planning to add a special silicone cover for the motor that will damp the vibrations.

Two words about focus gears reliability. In fact I do not see any wear of the worm or wheel, because after proper UV curing this type of plastic becomes very strong. But I plan to make a continues stress test to define the possible wear level. Making this complex parts on SLA printer is quite cheap and at the same time  quality and reliability is high, that's why I'm planning to use this solution in release design. I have a small experienсe with SLA printing, that's why feel free to point out flaws.

Some common view:

[attach=12][attach=13][attach=14][attach=15][attach=16][attach=17][attach=18]

Of course, all plastic parts STEP models will be avaliable.



P.S. no video yet, but maybe you can enjoy animation :)

[attachimg=19]
« Last Edit: July 22, 2020, 03:46:50 pm by VGN »
 
The following users thanked this post: Fraser, lukier

Offline Ismsanmar

  • Contributor
  • Posts: 21
  • Country: es
Perhaps the vibrations come from the lack of lubrication of the gears, apart from the slightly rough surface finish. You should apply some type of non-hygroscopic based grease, such as PTFE grease.
 

Online Fraser

  • Super Contributor
  • ***
  • Posts: 9543
  • Country: gb
This is the most impressive thermal camera project that I have seen to date. You have manufactured a very professional mechanical and electronic solution. If I was working on your PCB’s I would honestly think that they were the product of one of the well known thermal camera manufacturers  :-+ Most impressive  and I can only dream of having such design skills  :-+ :-+

Fraser
« Last Edit: July 22, 2020, 09:22:10 pm by Fraser »
 
The following users thanked this post: VGN

Offline Vipitis

  • Frequent Contributor
  • **
  • Posts: 675
  • Country: de
  • aspiring thermal photography enthusiast
That is so beautifully engineered, my plans for a 3D printed follow focus (external) that I sketched out during some downtime at an exam look pathetic.

I will take a deep look at this completely project and see which parts I can somehow use for my project. Experience and tools is what I feel I am lacking.
 
The following users thanked this post: VGN

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Perhaps the vibrations come from the lack of lubrication of the gears, apart from the slightly rough surface finish. You should apply some type of non-hygroscopic based grease, such as PTFE grease.
When I disassembled this motor, I found out that there was no any grease at all in the gearbox. I tried to apply a bike grease (the only grease I had), but things got worse, the "no load" current increased. Looks like too much grease in the gearbox prevented normal rotation, amount of lubrication must be proper. On the other hand, by default at max voltage (3.0V) the shaft rotates at 240RPM, so with ratio 1:136 we have motor rotating at about 32640 RPM, this is a lot. I suppose that probably vibration is caused by rotor disbalance.

This is the most impressive thermal camera project that I have seen to date. You have manufactured a very professional mechanical and electronic solution. If I was working on your PCB’s I would honestly think that they were the product of one of the well known thermal camera manufacturers  :-+ Most impressive  and I can only dream of having such design skills  :-+ :-+
Many thanks for your support! :)

That is so beautifully engineered, my plans for a 3D printed follow focus (external) that I sketched out during some downtime at an exam look pathetic.
Thank you! Just keep up. If your design or piece of a code starts looking pathetic for you, that means your skills are growing, this is a good sign. :-+ Wish you luck with your project! ;)

I will take a deep look at this completely project and see which parts I can somehow use for my project. Experience and tools is what I feel I am lacking.
Feel free to use any piece of this design. I'm going to start commiting some sources by the next week. I'll leave a link to my github repo.
 
The following users thanked this post: zrq

Offline dnhkng

  • Contributor
  • Posts: 11
  • Country: de
Soooo, VGN is the initials of three people, right? Vince, Gary, and Nancy? Because this project seems to be the work of a team of hardware designers, FPGA engineers, and electronics engineers all working together  :-+

Really impressive work, bravo!
 

Offline GeorgeOfTheJungle

  • Super Contributor
  • ***
  • Posts: 2689
  • Country: tr
 

Offline Flanbix

  • Contributor
  • Posts: 30
  • Country: gb
  • If you don't know, ask. If you know, share.
This project is great, lots of features. It is looking quite professional !  :-+
Keep the image and details comming.
 

Offline ajw107

  • Newbie
  • Posts: 4
  • Country: gb
Just wanted to say I joined this forum just to say thank you, and how impressed I am with your design.  I can;t wait until we can order the components.  Just got myself a new Audi 4GO980552A camera off eBay for £295 so I can do this project.  Should be buy the FPGA now as well, or will you be selling all the circuit boards as a kit?
If any one is interested in getting the same camera, there is one left at that price (less than a half the normal price for a used one) and the link is:
https://www.ebay.co.uk/itm/293490373204
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Soooo, VGN is the initials of three people, right? Vince, Gary, and Nancy? Because this project seems to be the work of a team of hardware designers, FPGA engineers, and electronics engineers all working together  :-+
Really impressive work, bravo!
Thank you! Hahaha, not Gary, me and Nancy call him Gabe ;)

This project is great, lots of features. It is looking quite professional !  :-+
Keep the image and details comming.
Thanks!

Just wanted to say I joined this forum just to say thank you, and how impressed I am with your design.  I can;t wait until we can order the components.  Just got myself a new Audi 4GO980552A camera off eBay for £295 so I can do this project.  Should be buy the FPGA now as well, or will you be selling all the circuit boards as a kit?
If any one is interested in getting the same camera, there is one left at that price (less than a half the normal price for a used one) and the link is:
https://www.ebay.co.uk/itm/293490373204
Thank you! I'm really very glad, that you and other people are interested in this project, that additionally motivates to continue. Yes, I have plans to make a some kind of kit, that anyone can buy. This kit will include everything you need, except the thermal camera, sd card, battery (due to some problems with air transportation of batteries). Later I will determine the exact kit contents. There is no need to buy a FPGA for this project. The design of this camera is developed in a way that you will not need even a solder iron to put all parts together. I will also make efforts to keep the functionality and price competetive with other brands, though that will be not easy.
« Last Edit: July 26, 2020, 09:14:57 am by VGN »
 

Offline zrq

  • Contributor
  • Posts: 25
  • Country: cn
This project is one of the most exciting projects I seen on this forum, I never expected a personal project can be so complete and elegant.
I'm considering buying a second hand BMW camera for the FPA, I wonder is there any pitfalls to avoid?
I would be also quite interested in the kit (given the price is reasonable  ;) ), as I don't have the equipment and experience to work on BGA packages.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
This project is one of the most exciting projects I seen on this forum, I never expected a personal project can be so complete and elegant.
I'm considering buying a second hand BMW camera for the FPA, I wonder is there any pitfalls to avoid?
I would be also quite interested in the kit (given the price is reasonable  ;) ), as I don't have the equipment and experience to work on BGA packages.
Thank you!  :)
Concerning pitfalls, I have described some of them in my previous post:
https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/msg3134494/#msg3134494
Actually, if the front window is not cracked and no one disassembled the camera, it should be in a good condition. If the window is cracked, the probability of sensor and lens survival is still quite high, but of course not 100%. As you can see two of my cameras with cracked windows have a good condition of lens and the sensor is working too. It's mostly a matter of price and luck. You can buy a new one for 300-500$ or take a chance with <100$ camera with cracked window. I bought two with cracked windows only because the price was low, I didn't expect that they will be working.

Concerning the kit, I believe you will not need to have any extra tools or skills to build and even calibrate the camera yourself.
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Here some more images, captured right by camera. Original images were in .bmp format, as this is the easiest way for image capturing, but I had to convert it to .png, as eevblog forum do not support .bmp attaches. Of course in future I plan to add png support right on camera, as bmps are much larger.

P.S. videos are comming soon!
« Last Edit: July 27, 2020, 07:26:11 pm by VGN »
 

Online Fraser

  • Super Contributor
  • ***
  • Posts: 9543
  • Country: gb
You cannot beat a decent microbolometer coupled with a good lens  :-+

At least you get a decent, low noise, image to process :) Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

Fraser
 

Offline ajw107

  • Newbie
  • Posts: 4
  • Country: gb
Thank you! I'm really very glad, that you and other people are interested in this project, that additionally motivates to continue. Yes, I have plans to make a some kind of kit, that anyone can buy. This kit will include everything you need, except the thermal camera, sd card, battery (due to some problems with air transportation of batteries). Later I will determine the exact kit contents. There is no need to buy a FPGA for this project. The design of this camera is developed in a way that you will not need even a solder iron to put all parts together. I will also make efforts to keep the functionality and price competetive with other brands, though that will be not easy.

That is great news, as I'm no good with a soldering iron (rubbish eyesight), although i will keep on trying to give it a go, much to the deprement of my fingers.  I have to admit, I have also just gone and bought a new SLA printer (Kelant S500, bit of a gamble as no reviews - but on paper is sounds good for the price), as my old ANET8 has long since given up the ghost and I've been itching to try an SLA for years.  The photos are great by the way, is that a brain cross-section in photos 11-14? I know it can't be, but it looks like it, he he!  Looking forward to being able to order the curcuitrs when they become available, so excited, like a kid with a new toy.
 

Offline ajw107

  • Newbie
  • Posts: 4
  • Country: gb
Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
Got my Autoliv NV3 today, does anyone know of a guide to hook it up to my PC or a Raspberry Pi?  I've tried googling, but everything I get is on the NV2 (or saying the NV3 isn't possible, which it must be now obviously).  I have to admit to being out of my depth with automotive system, and don't even recognise the connector (cirlcular with 4 sprung connectors inside).
This thread is actually the guide. :D
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...



You cannot beat a decent microbolometer coupled with a good lens  :-+
At least you get a decent, low noise, image to process :) Struggling with what the FLIR Lepton and Seek Cores produce is no fun at all.

The lens definitly matters. I don't have Flir E8 right now, but I remembered that earlier I have made a some photos at home. Here is the same bathroom with OpenIRV and FLIR E8 (hacked E4). Of course, shooting time and some other conditions are different:
 

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
I'm sorry, something completely wrong happens with image attachment on this forum. I had to delete and rewrite this message to fix images...



Earlier I showed some images, shooted in good conditions, when the scene tempretature difference is high. As you could see, there is no noticeable noise.
Now, let's look at abilities of this camera in poor conditions.

First, so that you can better understand what's going on, here is a photo of bicycle at the wall with a cup of warm water. The room temperature is +25.5 °С , the water in cup is +29.5 °С (measured by external thermometer). Temperature difference of the coldest and hottest objects is about 4-5°С. As you can see, the image is quite good, further we will analyze this scene without a cup of warm water.

[attachimg=6]



The same scene without a cup of warm water. This gif is a sequence of images shooted with a small time gap in a raw mode (no any kind of filtering):

[attachimg=1]

As you can see, the noise gets out, as temperature range is probably about ~1°С. The noise pattern is complex and quite specific. After reading tons of literature and some analysis and I came to think that there are two main noise components:
1. Row noise. This is a temporal noise, i.e. random fluctuations of the whole pixel line, that randomly changes every moment.
2. Column noise. This is a FPN (fixed pattern noise) that changes quite slowly and cannot be fully removed even by FFC (flat field correction) procedure.



Earlier I have developed a special averaging module that helps to average a sequence of multiple frames comming out of the sensor. Max possible number of averaged frames is 262144, though reasonable values are 2, 4, 8, 16, 32, 64, 128. Signal averaging is a common technique, intended to increase the strength of a signal relative to noise. So, let's try to use this module with different averaging values:

[attachimg=2]

We can see that starting from x8 value the row noise disapears, that confirms the nature of row noise. At the same time column noise is still present, averaging do not help to neutralize this type of noise, we need something more.



Few days ago I found a way to remove this column FPN in frequency domain by FFT (fast fourier transform) and a custom filter. As a test scene I decided to use some uniform object. Here you can see how it works. Image from camera (uniform body) is on the left, FFT of this image is on the right. Black lines on the FFT plot are the actual filter function, that "cuts out" the noise components. Here is the link to the online FFT tool that I used for experiments: https://www.ejectamenta.com/Fourifier-fullscreen/

[attachimg=3]



At last I decided to test both averaging and FFT filters at the complex scene with a lot of details:

[attachimg=4]

Also keep in mind that this experiments are done on equalized 8-bit image. I think that final image quality will be a little better if I apply FFT filter to the raw 14-bit pixel frames.



I don't want to start developing this FFT filter right now, it will be easier to develop it for new hardware and new FPGA. Though averaging is a good filter, it reduces the frame rate, that's why I should implement something else for row noise, of maybe FFT will be enogh, the time will show. At least I know how to deal with this noise. If you have good ideas and suggestions for image filtering, please don't keep in secret ;)



P.S. This is how lepton 80x60 "sees" this bicycle:  ;D

[attachimg=5]
« Last Edit: July 30, 2020, 11:27:18 am by VGN »
 

Offline zrq

  • Contributor
  • Posts: 25
  • Country: cn
For the similar noise I noticed on Xtherm cameras, I wrote a routine to reduce the single column noise by filtering in the temporal space.

Code: [Select]
merit = np.abs(cv2.Laplacian(cv2.GaussianBlur(imgFlt, (3, 3), 3), cv2.CV_32F)) / 65536
merit = 1 / pow(merit + 0.005, 3)
leftErr = imgFlt[:, 0:leftHalf] - cv2.GaussianBlur(imgFlt[:, 0:leftHalf], (1, 5), 10)
horiLeftAcc = np.average(leftErr, weights=merit[:, 0:leftHalf], axis=1)
rightErr = imgFlt[:, leftHalf:] - cv2.GaussianBlur(imgFlt[:, leftHalf:], (1, 5), 10)
horiRightAcc = np.average(rightErr, weights=merit[:, leftHalf:], axis=1)
vertErr = imgFlt - cv2.GaussianBlur(img, (5, 1), 10)
vertAcc = np.average(vertErr, weights=merit, axis=0)
for n in range(0, np.size(img, 0)):
    imgFlt[n, 0:leftHalf] = cv2.subtract(imgFlt[n, 0:leftHalf], (k[0] * horiLeftAcc[n])).squeeze(1)
    imgFlt[n, leftHalf:] = cv2.subtract(imgFlt[n, leftHalf:], (k[1] * horiRightAcc[n])).squeeze(1)
for n in range(0, np.size(img, 1)):
    imgFlt[:, n] = cv2.subtract(imgFlt[:, n], (k[2] * vertAcc[n])).squeeze(1)

However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked. :-\
 
The following users thanked this post: nikitasius

Offline VGN

  • Contributor
  • Posts: 25
  • Country: am
However, it looks like VGN's masking in the Fourier space did a better job. I tried similar method before, but it looks like I made a mistake by also masking the part close to 0 frequency and it never worked. :-\
I made the same mistake at first)
I also think that parametrizing this filter pattern like this will help to find optimal configuration:

[attachimg=3]



By the way, I found a cool Protoshop plug in for FFT noise reduction.
http://ft.rognemedia.no/

I tested it on bicycle image, drawing that cross filter pattern by hand. Now no image crop, no averaging, only pure FFT filter.

Before:

[attachimg=2]

After:

[attachimg=1]

You can see some luminance fluctuation in filtered image, I'm sure that this is caused by changes in histogram equalization, caused by noise peaks. That's why we should apply this filter on 14-bit raw image before histogram gain correction. Anyway, there is almost no high frequency row/column noise and that is really good!
« Last Edit: July 30, 2020, 11:03:38 am by VGN »
 

Offline ajw107

  • Newbie
  • Posts: 4
  • Country: gb
This thread is actually the guide. :D
Data lines of this 4 pin connector are going to MAX9259 IC: https://datasheets.maximintegrated.com/en/ds/MAX9259-MAX9260.pdf
This IC implements some proprietary gigabit link protocol...
You know, I thought that as soon as I pressed the post button (that's what I get for asking questions when I'm half asleep).  In my head there was some magical M12 CAN -> USB connector, he he!  Any news on when things will be ready for us to try (no rush, just over excitment, like a child with a new toy)!
 

Online Hydron

  • Frequent Contributor
  • **
  • Posts: 473
  • Country: gb
These noise issues are present on other microbolometers too - e.g. the ULIS 17u 640x480 FPA has a well known issue with column noise, and there are some filtering techniques that have been tried to improve it (some suggested by ULIS themselves I think). I'm not sure whether I have any info about these, but if I do and they are able to be distributed (unfortunately unlikely) I'll do so.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf