We also had to consider what to put on top of the silicon. We needed microlenses to improve light gathering, and we wanted the option of a Bayer mask for color imaging (we got triple resolution and quadruple sensitivity in monochrome, but almost everyone wanted color).
One more thing about Bayer filters: We also did research to see if we could tweak the sensor silicon and/or the Bayer color filters to get excellent color with less light lost in the filters. I attended a summer session at RIT's MCSL (Rochester Institute of Technology's Munsell Color Science Laboratory) to learn more about human color perception, the physics of light filtering and capture, and how color capture/rendering information is documented and shared (e.g., color profiles). We wanted to optimize our entire imaging process from subject to camera to display.
In the late '90's Carver Mead started Foveon to pursue an interesting property of silicon: Light of different wavelengths is absorbed at different depths. He designed the Foveon sensor to overcome the cruelest limitation of the Bayer filter matrix: The color information was captured with about 1/4 the resolution of the luminance (monochrome) information, causing significant spatial issues in color reconstruction. The Foveon sensor did this by vertically stacking the RGB pixels, with no color filtering needed above the sensor pixel.
In the Foveon sensor, the top photosite would capture "blueish" light (+ luminance), the middle would be "greenish" and the bottom would be "reddish". Unlike traditional color filters, silicon color filtering is extremely messy. But the Foveon team employed some amazing math to get good color with an imaging sensitivity only slightly below that of a monochrome sensor. But their sensor had other issues that took literally a decade to overcome, and it was relegated to niche applications.
We were very interested in the Foveon math. Could we modify our Bayer filter matrix to have it let more light through while still getting great color reconstruction? Could we do so well enough to eliminate the need for having a monochrome sensor option?
After all, the biggest problem in high-speed photography is capturing as many photons as possible as quickly as possible, with minimal losses along the entire optical path. So you start with the lens: There were several f 0.95 C-mount fixed lenses on the market, and even zooms around f 1.3.
Then you add AR (anti-reflection) coatings to the glass covering the sensor in its package. Multiple coatings are generally used, including UV and IR, though keeping losses down meant these coatings were very expensive.
You needed to put AR coatings on both sides of the glass: Light bouncing off the sensor (if any) must not be allowed to reflect into other pixels. The main visual effect of such reflection is reduced contrast. (There are specific test patters and exposure techniques that can reveal this behavior: You brightly illuminate a single pixel then examine the scatter in neighboring pixels.)
Finally you get to the sensor itself. First comes the microlenses. These are generally spherical (well, hemispherical), but we developed a very slick way to make non-spherical microlenses that captured a tiny bit more light.
You carefully do all the above work to lose as few photons as possible, then you intentionally discard over half of them in the next layer, the Bayer filter matrix. This is a knife to the heart of high-speed camera designers. While we all love looking at vibrant color images, we also mourn the photons that died just before reaching the silicon.
We successfully created prototype sensors with Bayer color filters having wider passbands, and obtained terrific color reconstruction results. The only problem was sensor yield: Our new filters had trouble surviving the rigors of semiconductor processing and packaging, and didn't survive at all well at the higher ends of our required operating temperature range. It was chemistry that finally made us revert to traditional and proven color filter materials for the HG-100K sensor.
To complete the story of the photon's path, the photons surviving the Bayer filter penetrate the silicon surface, where they are absorbed. This absorption can occur via multiple processes, but the end goal is always to have each photon dislodge one or more electrons. These electrons are accumulated as a charge until the exposure ends, after which the electrons are read out as a current, which one or more transistors at the photosite boost before it is read from the sensor.
The physics of photon capture can get hairy at high speed, since not all absorbed photons cause "prompt" electron release. This means the pixel must be forcibly reset before starting the next exposure, which can increase the dead-time between exposures, which in turn can decrease the overall frame rate. There are several tricks that can be used to limit this behavior, but it is always present to some extent.
There are multiple sources of "stray" or "non-photo emission" electrons within a sensor pixel. Most are due to thermal effects, which are seldom a problem in high-speed photography, but cause massive headaches for long-exposure astronomical photography (and hence their use of chilled sensors).
The most insidious source of stray electrons is the so-called "dark current", which is simple leakage into the photosite from nearby areas in the silicon, generally due to reverse-bias leakage. It accumulates in the photosite from the end of the reset until the end of the next exposure. If you put on the lens cap and take some images, you will see that none of the pixels are truly dark: All will have varying levels of electrons present even when no light is present.
Fortunately, it is straightforward to compensate for many stray electron sources by taking dark images just before each high-speed run, then processing the resulting video to remove the dark current's effect. I strongly recommend doing this over about 2K FPS: Your processed video will look much nicer. At slower speeds the photoelectrons will dominate over most noise sources, so there is little need for correction.
The dark current problem is enormously worse for IR cameras, which typically include an internal shutter that automatically activates every 20-60 seconds to quickly capture a fresh background image. You can see this happen in when Dave uses his FLIR camera.
So, when you follow a photon from the light source to the subject, which reflects it toward the camera lens, through optically coated glass, then through microlenses and Bayer arrays, into the silicon, conversion to electrons, readout to an ADC, storage in memory, downloading from the camera, then post-processing and finally being routed to a PC's video card and to a monitor, where a backlight shines through more color filters before being passed or blocked by the LCD layer and polarizing filters, it should become clear that there are many places for things to go wrong, and many places where corrections can be obtained and applied.
It's not just the camera: Only when the entire imaging process is optimized end-to-end can the best results be achieved. The camera may be the most important part of the process, but it doesn't stand alone. It takes time to learn how to properly use a high-speed imaging system.
But you can't do much about those awful losses in the Bayer filters. Sigh.
Home
Help
Search
Login
Register
