Open source CSI-2 Rx core for Xilinx FPGAs

[daveshah]

I'm not sure if there's much free out there (except by Lattice, and I'm not sure if that's really open source) for interfacing with common MIPI CSI-2 cameras, so I thought I'd share this project of mine. I'm currently using it with an Omnivision OV13850 to get 4k at 24/30fps; the CSI-2 receive part works well but the image processing (debayering etc) is still a work in progress.

It should be fairly easy to port it to other cameras - all timing parameters are VHDL generics, even up to 4k at 60fps, although anything other than 4-lanes and 10-bit RAW format would require new code. For improved timing performance it can output up to 4 pixels per clock; at 4k30 I'm using 2 pixels per clock on a Kintex-7 which reduces the amount of pipelining needed in the processing blocks.

https://github.com/daveshah1/CSI2Rx

[Scrts]

Very nice! I wish I had time to play with video stuff on FPGA. Some time ago I've tried Northwest Logic CSI-2 RX core, but it was locked all the way around.

[malkauns]

Hi Dave,
    You may remember me from the OV5640 thread.  Thanks for your help there although I was unsuccessful at parsing data correctly from the sensor.  I have bought the same OV13850 sensor that you have in the hopes that I can get it working using your code but I have some questions.  I am using the following pinout from the firefly website:



I am providing the stated voltages to pins 2, 3, 4 and 7 without any delay between them.  Are you doing something different here for the power up sequence?  I know that the datasheet says something about this but is it absolutely necessary?  If so how are you doing it?  I have used your code to provide the register values via i2c but am not able to obtain a clock signal.  Thanks in advance for any help.

[daveshah]

I didn't bother with power supply sequencing, all the rails would come up together in my system. Beware that despite that diagram pin 14 is a clock INPUT to the camera (nominally 24MHz), and if set up correctly the MIPI data clock will come out of CLKP/CLKN. Can you also confirm the status of other pins (CIF_RST and CIF_PDN0) - I believe the camera may need a reset before it can be used.

[malkauns]

Thanks, I'm getting a clock now.  Should I expect SOT's on all lanes at once?

[daveshah]

Yes, certainly within a clock cycle of each other. If you're not getting them I'd suspect an input delay or signal integrity issue.

[malkauns]

Thanks for your lightning fast reply.  Quick question.  In your code (csi_rx_top) where does pixel_clock_in come from?  Also am i correct in assuming that I just need to give a 200mhz clock to ref_clock_in?  I will start trying to get your code to work first before writing mine.

Thanks.

[daveshah]

pixel_clock_in should be equal to the output pixel clock and would be generated by a PLL in your design. Note that as the design outputs two pixels per clock it should be half the "actual" pixel clock. The PLL should be fed from the same ultimate source as the MCLK going into the camera so there is a constant relationship between the two.

Yes ref_clock_in is a 200MHz clock input for the IDELAYCTRL

[malkauns]

Ok, I'm still not getting anything to work although I am getting a byte clock of around 18.75MHz when I supply 12.5MHz to CIF_CLKOUT.  I also get an HSYNC of 12375Hz from video_hsync (in csi_rx_4lane).  I am not getting any SOT's.  This leads me to believe that there is perhaps something wrong with my physical setup.

This is what my setup looks like with my Zybo board:



and here is my xdc file:

Code: [Select]

set_property PACKAGE_PIN U15 [get_ports {dphy_clk[0]}]
set_property PACKAGE_PIN U14 [get_ports {dphy_clk[1]}]
set_property PACKAGE_PIN T15 [get_ports {dphy_d0[0]}]
set_property PACKAGE_PIN T14 [get_ports {dphy_d0[1]}]
set_property PACKAGE_PIN Y19 [get_ports {dphy_d1[0]}]
set_property PACKAGE_PIN Y18 [get_ports {dphy_d1[1]}]
set_property PACKAGE_PIN T10 [get_ports {dphy_d2[0]}]
set_property PACKAGE_PIN T11 [get_ports {dphy_d2[1]}]
set_property PACKAGE_PIN W19 [get_ports {dphy_d3[0]}]
set_property PACKAGE_PIN W18 [get_ports {dphy_d3[1]}]
set_property PACKAGE_PIN T20 [get_ports {CIF_CLK_OUT}]
set_property PACKAGE_PIN V18 [get_ports {ov_PWDN}]
set_property PACKAGE_PIN V17 [get_ports {ov_RET}]
set_property PACKAGE_PIN R14 [get_ports {ov_SIOD}]
set_property PACKAGE_PIN P14 [get_ports {ov_SIOC}]

set_property IOSTANDARD LVDS_25 [get_ports {dphy_clk[1]}]
set_property IOSTANDARD LVDS_25 [get_ports {dphy_d0[1]}]
set_property IOSTANDARD LVDS_25 [get_ports {dphy_d1[1]}]
set_property IOSTANDARD LVDS_25 [get_ports {dphy_d2[1]}]
set_property IOSTANDARD LVDS_25 [get_ports {dphy_d3[1]}]
set_property IOSTANDARD LVCMOS25 [get_ports {CIF_CLK_OUT}]
set_property IOSTANDARD LVCMOS25 [get_ports {ov_PWDN}]
set_property IOSTANDARD LVCMOS25 [get_ports {ov_RET}]
set_property IOSTANDARD LVCMOS25 [get_ports {ov_SIOD}]
set_property IOSTANDARD LVCMOS25 [get_ports {ov_SIOC}]
Also here's a pinout diagram for the Zynq that I'm using from a Xilinx doc:



I tried to attach all of the MIPI pins to clock capable pins (MRCC/SRCC) on the Zynq but not enough of them are available on the Zybo board so I did what I could.  Do you think that there is something wrong with the way I have connected things up?  Also, what hardware are you using where you have it working?

Thanks.

[daveshah]

If you're using my config unmodified, I'd expect a significantly higher byte clock on the order of 40-60MHz, (I can't remember the exact setup) with an input of 12.5MHz - assuming it works properly at that frequency. How are you measuring frequency? The byte clock is not continuous so counting cycles in a given time won't work. Can you post a scope signal of the byte_clock signal?

MIPI CSI-2 is pretty picky about signal integrity, I suspect the low signalling levels don't help. The setup you have is far from optimal, I used a custom FMC breakout board (https://github.com/daveshah1/DSITx/tree/master/hardware/fmc-v1.2) to connect the cameras to the FPGA. For a quick prototype I'd suggest replacing the jumper leads with twisted pair, ideally individually shielded and grounded at each end, for example salvaged from an Ethernet cable. Length matching is also a good idea, ideally adjusting cable lengths to compensate if there are different lengths on the dev board.

Also try adding the following timing constraints to your XDC file

Code: [Select]

create_clock -period 2.500 -name csi -waveform {0.000 1.250} [get_ports {dphy_clk[1]}]
create_clock -period 2.500 -name csi2 -waveform {1.250 2.500} [get_ports {dphy_clk[0]}]
set_input_delay -clock [get_clocks csi] 1.000 [get_ports {{dphy_d0[0]} {dphy_d0[1]} {dphy_d1[0]} {dphy_d1[1]} {dphy_d2[0]} {dphy_d2[1]} {dphy_d3[0]} {dphy_d3[1]}}]
set_input_delay -clock [get_clocks csi] -clock_fall 1.000 [get_ports {{dphy_d0[0]} {dphy_d0[1]} {dphy_d1[0]} {dphy_d1[1]} {dphy_d2[0]} {dphy_d2[1]} {dphy_d3[0]} {dphy_d3[1]}}]

Only the clock lane needs to be connected to a clock capable IO, although it shouldn't matter if others are too.

[malkauns]

Ok, some progress was made it seems.  Actually I had made a mistake when I modified some of your code and had commented out the 2nd last register setting in ov13850_4k_regs.  Now I am able to get a 74MHz byte clock when I supply 24MHz to CIF_CLKOUT.  Also video_valid is always high.  Am i correct in assuming that it is correctly obtaining data from all 4 lanes from the camera?  To answer your question about measuring frequency, I am just measuring rising edges per second.  Unfortunately I don't have anything as extravagant as a scope right now.  My next question is, if this is actually working (and I hope to God it really is!!) then is a video_hsync of 11880Hz and video_vsync of 3Hz what you would expect with your unmodified setup?  3Hz sounds a little low to me.  Thanks.

[daveshah]

I can't guarantee video_valid works properly, IIRC there may be some issues with it - checking the actual data coming out is the only way to guarantee it's working. 74MHz byte clock seems low with 24MHz in but might be OK if you're counting edges due to it being discontinuous.

hsync and vsync will free run based on the pixel clock but should sync to the camera once valid data is appearing. What pixel clock are you feeding in, and what are your timing settings? I can't think how the mechanism that synchronises vsync to the camera would ever produce a significantly lower framerate than expected.

[malkauns]

Thanks for drawing my attention to the pixel clock.  I didn't realize that it would have that kind of effect (I'm still new to all this video stuff).  I guess I had it set too low at 26MHz (connected to the same clock source for CIF_CLKOUT).  It seems that the pixel clock is determining the frame rate.  With a pixel clock of 162.5MHz I get a video_hsync of ~80445Hz and a video_vsync of ~25Hz translating to roughly 3217 lines per frame.  So right now my configuration is:

Input:
pixel_clock_in: 162.5MHz
ref_clock_in: 200MHz
CIF_CLKOUT: 26MHz

Output:
byte_clock_out: 81MHz
video_vsync: 25Hz
video_hsync: 80445Hz

Does this sound like what you would expect?  If so, I guess I should start looking at the image data.  When exactly do I sample video_data?  Every byte_clock_out cycle after video_vsync and video_hsync go high?

[daveshah]

video_data is sampled based on the pixel clock, and is valid whenever video_den is high. Note that two pixels per clock are output. The pixel_clock for the default config should be 145MHz, with an MCLK of about 24.4MHz. As a quick test you could output the current pixel value to the integrated logic analyser or some other output device and confirm it decreases when you cover the camera.

[malkauns]

Yep, I've integrated the simple_debayer code into csi_rx_top and verified that when I shine my camera flash into the OV13850 lens, the video data (RGB at this point) is mostly 0xffffffff.  I added some quick code to count the amount of times video_den is high on each rising edge of pixel_clock_in.  This comes to roughly 102MHz.  Am I correct in assuming that the first RGB pixel of a frame occurs when pixel_clock_in, video_den, video_vsync and video_hsync are all high?  Then only read (2 pixels) at every pixel_clock_in, video_den until I have obtained 3840 pixels.  Repeat when those 4 signals are high again for each video line?

[daveshah]

Actually as per standard timing conventions with the default configs the vsync pulse will be sometime before the first line and hsync pulse sometime before the first pixel. Otherwise that all looks good.

[malkauns]

Ok, I'll experiment with this (probably with more questions ).  Thanks for your fast reply as always and all your help so far.  Very much appreciated!!

[malkauns]

Hi Dave, made some progress but just need a little help in getting a valid image.  I have turned on the color bar option (which I assume is a static image, correct?) by setting register 0x5e00 to 0x80.  However, I get an image like this:



Here's a shot of the top left of that image:



I think the problem may lie in my lack of understanding of video_data_even and video_data_odd.  Am i correct in assuming that video_data_even is pixel 0, 2, 3, 4 etc. and video_data_odd is pixel 1, 3, 5, 7 etc. ?  Either that or there's something wrong in my code logic.

[cirthix]

I was under the impression that the signaling levels in the mipi D-PHY were incompatibile with the inputs of most fpgas out there, including artix/zynq/kintex families.

There are hardware workarounds, but it looks like you're trying to hook it up directly to the pins.

[malkauns]

I was under the impression that the signaling levels in the mipi D-PHY were incompatibile with the inputs of most fpgas out there, including artix/zynq/kintex families.

There are hardware workarounds, but it looks like you're trying to hook it up directly to the pins.

I also wondered if that was the case when I started this project but everything seems to sync up.  The vsync and hsync MIPI signals are making sense in my case and I am obtaining 23 vsyncs/sec and 2160 hsyncs/sec.  Between each hsync i'm obtaining exactly 3840 pixels.  I don't think this would be the case if D-PHY signal incompatibility was an issue.  There's either something wrong in my code logic or I am not understanding things correctly.  Having said that, what are the hardware workarounds that you mention?

[daveshah]

Have been able to get completely stable performance using LVDS inputs provided signal integrity is good.

I think the problem in your case is the input "skew" settings need tweaking slightly to get the best possible performance.

Actually the way the vhdl is implemented valid video timings will likely (except vsync) will be generated even if the received input is bad.

[cirthix]

I was under the impression that the signaling levels in the mipi D-PHY were incompatibile with the inputs of most fpgas out there, including artix/zynq/kintex families.

There are hardware workarounds, but it looks like you're trying to hook it up directly to the pins.

I also wondered if that was the case when I started this project but everything seems to sync up.  The vsync and hsync MIPI signals are making sense in my case and I am obtaining 23 vsyncs/sec and 2160 hsyncs/sec.  Between each hsync i'm obtaining exactly 3840 pixels.  I don't think this would be the case if D-PHY signal incompatibility was an issue.  There's either something wrong in my code logic or I am not understanding things correctly.  Having said that, what are the hardware workarounds that you mention?

Check out xapp 585 for per-bit deskewing and xapp 894 for the d-phy stuff.

If you think your problem lies in the logic, I'd suggest using the integrated logic analyzers to look at a single line of data at a time compared to the raw input stream.

That said, I'm not sure how fast you're trying to run this, but the attachment from the fpga board to the fpc breakout board looks terrible (and that fpc breakout isnt going to help).

[daveshah]

In terms of signal integrity I would also recommend moving away from the breakout board and jumper cables if you haven't already.

The link is running at the best part of 1Gbit with low signal levels and quite tight timing requirements so you're lucky to be getting what you're getting really.

[malkauns]

hmm, what would you suggest?  I'm not sure exactly how else I would be able to attach it to my Zybo board.  I'm open to buying a new dev board though if its not too expensive.

[cirthix]

hmm, what would you suggest?  I'm not sure exactly how else I would be able to attach it to my Zybo board.  I'm open to buying a new dev board though if its not too expensive.

This should be fine

[malkauns]

Thanks for your insights cirthix and Dave.  The fact that i'm getting an image such as the one below is very encouraging despite my horrendous setup.  From the look of the Firefly OV13850 board, I'm leaning toward making my own version that will connect directly to the Zybo board as suggested in cirthix's pic.  Upon looking at this board I see that there are 10 components (resistors/logic level converter?) on the front and 8 capacitors on the back.  As there are 10 on the front I'm guessing that they each sit between the MIPI data (8 pins) and clock pins (2 pins).  The capacitors are probably set up similar to my home made power board that you guys see in the pic.  Do you think that this is correct?  If so, what exactly are the 10 components that sit on the front of the board?

[daveshah]

I believe the 10 resistors in the MIPI lines are 0 ohm resistors, presumably for test or because they weren't sure if they needed some resistance for some reason. I wouldn't bother with them personally, and just connect the MIPI straight through. The capacitors will be for local decoupling, probably 1 or 10 uF.

[malkauns]

what do you mean by "local decoupling" ?

[Berni]

I been working with MIPI-DSI on the Lattice and i found signal integrity to be pretty important.

My MIPI bus was running at 600Mbit over a single lane there so pretty quick, but there are even faster MIPI buses out there. I was running it over ribbon cable with the differential pairs next to each other and there are cases where touching large areas of the cable could cause glitches in the image. Terminating the lines correctly made it more resilient. Touching just one of the pairs with a scope probe breaks it completely. I had to use a differential active probe to get a good look at it without mangling up the waveform (And used a 4GHz scope)

So do take lots of care with the wiring for the MIPI pairs. I am surprised it worked at all with your wire ratsnest setup.

The way to go is to try maintaining a differential transmission line for as long as possible. the FPC cable is a reasonable transmission line with the pins next to each other and so rounded by ground in the pinout. Then you want to use twisted pair to wire to your FPGA board. Most FPGA dev boards have some pins on the header that are routed deferentially to the FPGA, its easy to spot them on the PCB layout, use those if possible. Also one ground is not enough, use 3 or 4 ground wires to make sure you have a low inductance connection. For power i would not be worried too much, as long as you have a capacitor close to the camera connector.

[alga_kbk]

Hi there!

Thanks, Dave for your source codeshare!

I'm new in image processing sorry if I bore you. I'm also trying to set up is the project with the OV13850 and have the question, on I2C is there a way to obtain the response of camera?  In order to be sure that camera is working. As you all mentioned before about signal integrity, I haven't already made breakout board for connection just using the cable for connection and didn't obtain anything on MIPI bus

[daveshah]

I don't implement such a thing in my simple I2C core, but you could put something like a soft processor with a I2C core in to implement that and do further testing.

The code as is will run at a line rate of about 1Gbps/lane, this will almost certainly not work using anything other than a PCB. Can you post a photo of your setup?

[alga_kbk]

Hi Dave!

Thanks for your help! Sure will attach the setup photo later. As I understand on I2C there is a response from the camera after every message that it obtained, will try to play with that. I wanted to know, is there any way to decrease the MIPI line rate to 100Mbps/lane?

[daveshah]

Hi alga!

It's a long time since I've touched the config for an Omnivision camera, but it should be possible to reduce the line rate considerably. The first thing to try is to reduce the input clock (MCLK) to the camera, it should be possible to reduce that by at least a factor of two. Beyond that you'll probably have to play with the PLL and D-PHY settings. Maybe have a look at existing configs in Linux kernel drivers for your camera?

Which camera module are using again?

[alga_kbk]

Hi Dave!

Thank you very much, today will play with it. I'm using  OV13850 camera from Firefly RK3288 module.