Image Sensor FPGA Interface.

[VintageTech]

For the past couple days I have been beginning a project to create a camera based around an OnSemi MT9P031 5MP image sensor. I dont want to use a sensor module because most are built for video and I'm looking to do both. I have quickly come to the conclusion that an Arduino is laughably underpowered for this application. I need an ARM based micro or an FPGA to do it effectively.

With the background out of the way, I have a few questions

1) FPGA interface
What is the best device to use in the sensor interface? I need to be able to take images, Dump the Raw Files to an SD Card and then edit later. A built in Display would be very nice, however, It is supposed to be a film camera like device so not necessary.

2) how does one go about image processing in an FPGA? I'm not very familiar with FPGA programming but I do have a solid background in C/C++ and I am willing to learn for the project.

Thank you all,
-Vintage Tech

[nctnico]

What you need is a board with a SoC (system on chip) which runs Linux. Raspberry Pi might be worth looking into if this is a hobby project.

[Someone]

What you need is a board with a SoC (system on chip) which runs Linux. Raspberry Pi might be worth looking into if this is a hobby project.

That might be the broadest and least useful comment I've seen here for a while, linux is irrelevant to the needs of getting frame data from a video sensor like this. It might be nice to have but without a suitable physical interface there is no work to do.

For the past couple days I have been beginning a project to create a camera based around an OnSemi MT9P031 5MP image sensor. I dont want to use a sensor module because most are built for video and I'm looking to do both. I have quickly come to the conclusion that an Arduino is laughably underpowered for this application. I need an ARM based micro or an FPGA to do it effectively.

With the background out of the way, I have a few questions

1) FPGA interface
What is the best device to use in the sensor interface? I need to be able to take images, Dump the Raw Files to an SD Card and then edit later. A built in Display would be very nice, however, It is supposed to be a film camera like device so not necessary.

2) how does one go about image processing in an FPGA? I'm not very familiar with FPGA programming but I do have a solid background in C/C++ and I am willing to learn for the project.

Thank you all,
-Vintage Tech

FPGAs are often used for camera interfaces as they can instantiate almost arbitrary data interfaces to the outside world, you can take a single FPGA and it will be able to connect to many different camera interfaces. Embedded (micro)processors/controllers might only have one or two different interfaces for cameras, most will have none. So you can either hunt for an embedded SoC platform that has a suitable interface and program on that, or program it all from scratch with an FPGA.

Pause for a moment here, and realise that you will need a much lower level of electronics and digital skills to get the FPGA route off the ground. If you've never done anything like it before you will spend a long time learning. It opens up many more options and is a useful skill to have, but its an investment for sure. Instead of buffering everything into frames and processing each step, then handing off to a new buffer for the next step to work on as is the naive software approach, FPGAs will process all the steps simultaneously and you will have minimal buffering between each stage. Memory space is at a premium on FPGAs so algorithms are implemented in radically different ways, and its all in fixed point.

Note also the camera you want to use will put out more than 100MB/s, which is hard to write to an SD card, you'll either need some image compression or use much less than the full bandwidth (frame rate * resolution).

[nctnico]

What you need is a board with a SoC (system on chip) which runs Linux. Raspberry Pi might be worth looking into if this is a hobby project.

That might be the broadest and least useful comment I've seen here for a while, linux is irrelevant to the needs of getting frame data from a video sensor like this. It might be nice to have but without a suitable physical interface there is no work to do.

No, given the requirements and how standard camera interfaces have become it is a very useful comment. The OP clearly wants to do something meaningfull with the data and just slapping a physical camera interface onto an FPGA is least of the problems. A reasonable SoC has hardware compression and thus can compress the video stream enough so an SD card can keep up. Also putting a display on the device is a no-brainer. Couple that with a decent processor core and you have a platform ready quickly. IF this is a hobby project the Raspberry Pi is probably the easiest solution on which (a well chosen) camera module or sensor chip, SD card and display work out of the box. Going for a SoC design is good to go for medium to high volume. An FPGA based design is probably the worst choice unless there is some niche data processing requirement which cannot be done with a quadcore ARM running at 1GHz and hardware video compression.

[Someone]

What you need is a board with a SoC (system on chip) which runs Linux. Raspberry Pi might be worth looking into if this is a hobby project.

That might be the broadest and least useful comment I've seen here for a while, linux is irrelevant to the needs of getting frame data from a video sensor like this. It might be nice to have but without a suitable physical interface there is no work to do.

No, given the requirements and how standard camera interfaces have become it is a very useful comment. The OP clearly wants to do something meaningfull with the data and just slapping a physical camera interface onto an FPGA is least of the problems. A reasonable SoC has hardware compression and thus can compress the video stream enough so an SD card can keep up. Also putting a display on the device is a no-brainer. Couple that with a decent processor core and you have a platform ready quickly. IF this is a hobby project the Raspberry Pi is probably the easiest solution on which (a well chosen) camera module or sensor chip, SD card and display work out of the box. Going for a SoC design is good to go for medium to high volume. An FPGA based design is probably the worst choice unless there is some niche data processing requirement which cannot be done with a quadcore ARM running at 1GHz and hardware video compression.

Can you not parse the context of the OPs question? They are asking about interfaces after realising their ideas will never work on an Arduino. Two well thought out replies about interfacing and your barely considered reply suggesting software as the answer.

[Rasz]

For the past couple days I have been beginning a project to create a camera based around an OnSemi MT9P031 5MP

is there something special about this sensor? how does Sony IMX219 sound?

What you need is a board with a SoC (system on chip) which runs Linux. Raspberry Pi might be worth looking into if this is a hobby project.

That might be the broadest and least useful comment I've seen here for a while, linux is irrelevant to the needs of getting frame data from a video sensor like this. It might be nice to have but without a suitable physical interface there is no work to do.

it would be ..until a year ago when 6of9 (former broadcom employee) wrote _universal_ MIPI CSI driver for the Pee
Its able to pump MIPI data frames to ARM memory, so as long as:
-this module has mipi interface
-you have enough documentation to set it up over i2c and make it start pumping out data
you are golden with ordinary RasPEE and a bit of C or even Python code.

[ebclr]

The  MT9P031 doe not need any interface to connect to a FPGA, only a buck of wires ( pcb's tracks ) and some resistors, since everything is digital 1.7 ..1.9V (1.8V nominal). Perhaps you are looking for IP to handle cameras not interface

[VintageTech]

Firstly, Thank you to everyone who has taken the time to respond, I am very grateful for all of the helpful tips provided!

A few points of clarification to add:

)1 I am looking to do still image capture as a primary function of the camera. Video is, at this point, something that would be nice to include, but its not a primary objective.

2) The reason that I went for the MT9P031 was due to the relative simplicity of this sensor. Its outputs aren't differential pairs and it has an I2C interface for control. I used this sensor as a starting point but I am willing to consider other options such as the Sony IMX219.

3) JPEG compression is another "nice to have" thing, However I shoot RAW on my camera so its not really a necessity. I'm really not sure what the RAW format is for this sensor but I hope to be able to configure it to output in something of the sort.

4) There is an open source cinema camera made by an organization called Apertusº which uses a CMV12000 sensor that works over LVDS. They are using two FPGAs as an interface to a MicroZed running a custom version of Magic Lantern. When Compared to cameras from companies like RED or Arri, this camera is actually quite cheap. One would assume that it should be possible to build one that uses a much lower end sensor to primarily capture stills

5) I am looking in to using the SoCs discussed above, Thank you all for your time!
A few links:
CMV12000 Datasheet:http://www.betopcom.com/public/images/products//20131126180222.pdfPHPSESSID=to1n9v3fgm9u08lskmupt9jm76

Apertusº: Apertus.org

[nctnico]

There is much more to making a camera than connecting an image sensor to a chip. Think about exposure time, (automatic) gain control, gamma correction, brightness, contrast, rolling shutter versus fixed shutter.

[VintageTech]

There is much more to making a camera than connecting an image sensor to a chip. Think about exposure time, (automatic) gain control, gamma correction, brightness, contrast, rolling shutter versus fixed shutter.

From what I have gathered, most mirrorless (think phone or micro four thirds) cameras handle these parameters with the I2C or SPI busses that control them. My intention was to use the SoC (or secondary MCU) to control the image capture from the sensor then dump it into the SoC to be written to the SD card.  Thus all factors are controlled in camera. Nothing that complex in terms of control

[nctnico]

In theory yes but getting the details in order to setup the camera may not be so easy. Even with support from Omnivision it took me a lot of time to get one of their highly integrated camera modules to work. And even then some things just didn't seem to work according to the documentation and on those occasions the support staff wasn't very helpful either (despite NDAs where put into place).

[Kilrah]

The I2C bus may be used to set the parameters in the sensor (exposure time, ISO...) but if you want auto exposure you still need to actually analyze the image data to know whether it's correct and figure out what adjsutments to do if it's not. That's where it gets VERY complex.

Some sensors include auto exposure facilties but the one you refer to doesn't.

If you make a purely manual camera and just dump raw data (DNG format?) you'd indeed be able to skip most complexities, although most cameras still do SOME amount of processing to RAW data e.g. removal of banding (line/column offset differences), bad pixel correction etc using factory calibration data.

[Rasz]

)1 I am looking to do still image capture as a primary function of the camera. Video is, at this point, something that would be nice to include, but its not a primary objective.

2) The reason that I went for the MT9P031 was due to the relative simplicity of this sensor. Its outputs aren't differential pairs and it has an I2C interface for control. I used this sensor as a starting point but I am willing to consider other options such as the Sony IMX219.

3) JPEG compression is another "nice to have" thing, However I shoot RAW on my camera so its not really a necessity. I'm really not sure what the RAW format is for this sensor but I hope to be able to configure it to output in something of the sort.

fantastic, start there:
https://learn.adafruit.com/diy-wifi-raspberry-pi-touch-cam/overview
https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=145815

pee + camera module will cost you ~$60 _total_ and work out of the box. This setup will let you learn what goes into interfacing modern camera module and deliver all 3 points you mentioned above.
hint: its not as simple as "oh its just i2c", sensor documentation is either non existent, secret under NDAs, buggy/misleading, or useless without huge body of domain specific knowledge (tuning color conversion from bayer).

[VintageTech]

)1 I am looking to do still image capture as a primary function of the camera. Video is, at this point, something that would be nice to include, but its not a primary objective.

2) The reason that I went for the MT9P031 was due to the relative simplicity of this sensor. Its outputs aren't differential pairs and it has an I2C interface for control. I used this sensor as a starting point but I am willing to consider other options such as the Sony IMX219.

3) JPEG compression is another "nice to have" thing, However I shoot RAW on my camera so its not really a necessity. I'm really not sure what the RAW format is for this sensor but I hope to be able to configure it to output in something of the sort.

fantastic, start there:
https://learn.adafruit.com/diy-wifi-raspberry-pi-touch-cam/overview
https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=145815

pee + camera module will cost you ~$60 _total_ and work out of the box. This setup will let you learn what goes into interfacing modern camera module and deliver all 3 points you mentioned above.
hint: its not as simple as "oh its just i2c", sensor documentation is either non existent, secret under NDAs, buggy/misleading, or useless without huge body of domain specific knowledge (tuning color conversion from bayer).

The RPi + camera option is definitely something I have/Will continue to look into. The issue here being that the RPi camera has a lens built in and I really don't want to use a rotary tool or other cutting device to get it off. I plan to use an existing lens (old dslr lenses specifically, I'm aware of the massive crop factor) platform and eventually expand this into using more advanced, Bigger image sensors (MIPI CSI and the like). Essentially I am looking for a hardware level starting point so I can get a clear understanding of what goes on in a digital camera, then expand it into an open sourced DSLR. Its really crazy, and I'm okay with that. To my knowledge the MT9P031 isn't protected under Secret NDA's (they give you I2C registers & values) so it seems to be an ideal starting point. EDIT: There is an RPI camera with a C mount that could be adapted to my needs. I'll look into using this as well, though the Rpi camera seems to have a bigger crop factor. Also literally hacking it apart isn't really a concern because the lens module can be lifted right out.

Where do you plan to source these sensors (if you do) in small quantities?  Are they available in low quantities from distribution with no problem?  Maybe I'll make a camera with some such sensor one day though I still have some older model sub-3MP CIS parts to use waiting for my free time but now there are much higher resolution / performance sensors like this than some of those older ones so maybe it is best to use something newer like this model when I get around to it.

The MT9P031 sensor is available for about 20 USD from most distributors in low quantity there are also 12+ megapixel versions that may use a different interface (will verify). It appears that there really isn't a problem acquiring the parts or the datasheet (with all relevant information I2C, Data rates, output format, ETC). Thanks for your replies!