Author Topic: Open source CSI-2 Rx core for Xilinx FPGAs  (Read 11397 times)

0 Members and 1 Guest are viewing this topic.

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Open source CSI-2 Rx core for Xilinx FPGAs
« on: November 16, 2016, 04:22:56 pm »
I'm not sure if there's much free out there (except by Lattice, and I'm not sure if that's really open source) for interfacing with common MIPI CSI-2 cameras, so I thought I'd share this project of mine. I'm currently using it with an Omnivision OV13850 to get 4k at 24/30fps; the CSI-2 receive part works well but the image processing (debayering etc) is still a work in progress.

It should be fairly easy to port it to other cameras - all timing parameters are VHDL generics, even up to 4k at 60fps, although anything other than 4-lanes and 10-bit RAW format would require new code. For improved timing performance it can output up to 4 pixels per clock; at 4k30 I'm using 2 pixels per clock on a Kintex-7 which reduces the amount of pipelining needed in the processing blocks.

https://github.com/daveshah1/CSI2Rx
 

Offline Scrts

  • Frequent Contributor
  • **
  • Posts: 729
  • Country: lt
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #1 on: November 16, 2016, 07:45:02 pm »
Very nice! I wish I had time to play with video stuff on FPGA. Some time ago I've tried Northwest Logic CSI-2 RX core, but it was locked all the way around.
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #2 on: May 22, 2017, 09:39:14 pm »
Hi Dave,
    You may remember me from the OV5640 thread.  Thanks for your help there although I was unsuccessful at parsing data correctly from the sensor.  I have bought the same OV13850 sensor that you have in the hopes that I can get it working using your code but I have some questions.  I am using the following pinout from the firefly website:



I am providing the stated voltages to pins 2, 3, 4 and 7 without any delay between them.  Are you doing something different here for the power up sequence?  I know that the datasheet says something about this but is it absolutely necessary?  If so how are you doing it?  I have used your code to provide the register values via i2c but am not able to obtain a clock signal.  Thanks in advance for any help.
 

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #3 on: May 23, 2017, 10:45:07 am »
I didn't bother with power supply sequencing, all the rails would come up together in my system. Beware that despite that diagram pin 14 is a clock INPUT to the camera (nominally 24MHz), and if set up correctly the MIPI data clock will come out of CLKP/CLKN. Can you also confirm the status of other pins (CIF_RST and CIF_PDN0) - I believe the camera may need a reset before it can be used.
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #4 on: May 24, 2017, 05:00:41 pm »
Thanks, I'm getting a clock now.  Should I expect SOT's on all lanes at once?
 

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #5 on: May 24, 2017, 05:06:02 pm »
Yes, certainly within a clock cycle of each other. If you're not getting them I'd suspect an input delay or signal integrity issue.
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #6 on: May 24, 2017, 05:27:41 pm »
Thanks for your lightning fast reply.  Quick question.  In your code (csi_rx_top) where does pixel_clock_in come from?  Also am i correct in assuming that I just need to give a 200mhz clock to ref_clock_in?  I will start trying to get your code to work first before writing mine.

Thanks.
 

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #7 on: May 24, 2017, 07:24:30 pm »
pixel_clock_in should be equal to the output pixel clock and would be generated by a PLL in your design. Note that as the design outputs two pixels per clock it should be half the "actual" pixel clock. The PLL should be fed from the same ultimate source as the MCLK going into the camera so there is a constant relationship between the two.

Yes ref_clock_in is a 200MHz clock input for the IDELAYCTRL
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #8 on: June 01, 2017, 12:32:41 am »
Ok, I'm still not getting anything to work although I am getting a byte clock of around 18.75MHz when I supply 12.5MHz to CIF_CLKOUT.  I also get an HSYNC of 12375Hz from video_hsync (in csi_rx_4lane).  I am not getting any SOT's.  This leads me to believe that there is perhaps something wrong with my physical setup.

This is what my setup looks like with my Zybo board:



and here is my xdc file:

Code: [Select]
set_property PACKAGE_PIN U15 [get_ports {dphy_clk[0]}]
set_property PACKAGE_PIN U14 [get_ports {dphy_clk[1]}]
set_property PACKAGE_PIN T15 [get_ports {dphy_d0[0]}]
set_property PACKAGE_PIN T14 [get_ports {dphy_d0[1]}]
set_property PACKAGE_PIN Y19 [get_ports {dphy_d1[0]}]
set_property PACKAGE_PIN Y18 [get_ports {dphy_d1[1]}]
set_property PACKAGE_PIN T10 [get_ports {dphy_d2[0]}]
set_property PACKAGE_PIN T11 [get_ports {dphy_d2[1]}]
set_property PACKAGE_PIN W19 [get_ports {dphy_d3[0]}]
set_property PACKAGE_PIN W18 [get_ports {dphy_d3[1]}]
set_property PACKAGE_PIN T20 [get_ports {CIF_CLK_OUT}]
set_property PACKAGE_PIN V18 [get_ports {ov_PWDN}]
set_property PACKAGE_PIN V17 [get_ports {ov_RET}]
set_property PACKAGE_PIN R14 [get_ports {ov_SIOD}]
set_property PACKAGE_PIN P14 [get_ports {ov_SIOC}]

set_property IOSTANDARD LVDS_25 [get_ports {dphy_clk[1]}]
set_property IOSTANDARD LVDS_25 [get_ports {dphy_d0[1]}]
set_property IOSTANDARD LVDS_25 [get_ports {dphy_d1[1]}]
set_property IOSTANDARD LVDS_25 [get_ports {dphy_d2[1]}]
set_property IOSTANDARD LVDS_25 [get_ports {dphy_d3[1]}]
set_property IOSTANDARD LVCMOS25 [get_ports {CIF_CLK_OUT}]
set_property IOSTANDARD LVCMOS25 [get_ports {ov_PWDN}]
set_property IOSTANDARD LVCMOS25 [get_ports {ov_RET}]
set_property IOSTANDARD LVCMOS25 [get_ports {ov_SIOD}]
set_property IOSTANDARD LVCMOS25 [get_ports {ov_SIOC}]

Also here's a pinout diagram for the Zynq that I'm using from a Xilinx doc:



I tried to attach all of the MIPI pins to clock capable pins (MRCC/SRCC) on the Zynq but not enough of them are available on the Zybo board so I did what I could.  Do you think that there is something wrong with the way I have connected things up?  Also, what hardware are you using where you have it working?

Thanks.
« Last Edit: June 01, 2017, 12:34:16 am by malkauns »
 

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #9 on: June 01, 2017, 10:55:18 am »
If you're using my config unmodified, I'd expect a significantly higher byte clock on the order of 40-60MHz, (I can't remember the exact setup) with an input of 12.5MHz - assuming it works properly at that frequency. How are you measuring frequency? The byte clock is not continuous so counting cycles in a given time won't work. Can you post a scope signal of the byte_clock signal?

MIPI CSI-2 is pretty picky about signal integrity, I suspect the low signalling levels don't help. The setup you have is far from optimal, I used a custom FMC breakout board (https://github.com/daveshah1/DSITx/tree/master/hardware/fmc-v1.2) to connect the cameras to the FPGA. For a quick prototype I'd suggest replacing the jumper leads with twisted pair, ideally individually shielded and grounded at each end, for example salvaged from an Ethernet cable. Length matching is also a good idea, ideally adjusting cable lengths to compensate if there are different lengths on the dev board.

Also try adding the following timing constraints to your XDC file

Code: [Select]
create_clock -period 2.500 -name csi -waveform {0.000 1.250} [get_ports {dphy_clk[1]}]
create_clock -period 2.500 -name csi2 -waveform {1.250 2.500} [get_ports {dphy_clk[0]}]
set_input_delay -clock [get_clocks csi] 1.000 [get_ports {{dphy_d0[0]} {dphy_d0[1]} {dphy_d1[0]} {dphy_d1[1]} {dphy_d2[0]} {dphy_d2[1]} {dphy_d3[0]} {dphy_d3[1]}}]
set_input_delay -clock [get_clocks csi] -clock_fall 1.000 [get_ports {{dphy_d0[0]} {dphy_d0[1]} {dphy_d1[0]} {dphy_d1[1]} {dphy_d2[0]} {dphy_d2[1]} {dphy_d3[0]} {dphy_d3[1]}}]

Only the clock lane needs to be connected to a clock capable IO, although it shouldn't matter if others are too.
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #10 on: June 04, 2017, 05:34:55 am »
Ok, some progress was made it seems.  Actually I had made a mistake when I modified some of your code and had commented out the 2nd last register setting in ov13850_4k_regs.  Now I am able to get a 74MHz byte clock when I supply 24MHz to CIF_CLKOUT.  Also video_valid is always high.  Am i correct in assuming that it is correctly obtaining data from all 4 lanes from the camera?  To answer your question about measuring frequency, I am just measuring rising edges per second.  Unfortunately I don't have anything as extravagant as a scope right now.  My next question is, if this is actually working (and I hope to God it really is!!) then is a video_hsync of 11880Hz and video_vsync of 3Hz what you would expect with your unmodified setup?  3Hz sounds a little low to me.  Thanks.
 

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #11 on: June 10, 2017, 02:45:12 pm »
I can't guarantee video_valid works properly, IIRC there may be some issues with it - checking the actual data coming out is the only way to guarantee it's working. 74MHz byte clock seems low with 24MHz in but might be OK if you're counting edges due to it being discontinuous.

hsync and vsync will free run based on the pixel clock but should sync to the camera once valid data is appearing. What pixel clock are you feeding in, and what are your timing settings? I can't think how the mechanism that synchronises vsync to the camera would ever produce a significantly lower framerate than expected.
« Last Edit: June 10, 2017, 02:47:11 pm by daveshah »
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #12 on: June 11, 2017, 07:47:57 am »
Thanks for drawing my attention to the pixel clock.  I didn't realize that it would have that kind of effect (I'm still new to all this video stuff).  I guess I had it set too low at 26MHz (connected to the same clock source for CIF_CLKOUT).  It seems that the pixel clock is determining the frame rate.  With a pixel clock of 162.5MHz I get a video_hsync of ~80445Hz and a video_vsync of ~25Hz translating to roughly 3217 lines per frame.  So right now my configuration is:

Input:
pixel_clock_in: 162.5MHz
ref_clock_in: 200MHz
CIF_CLKOUT: 26MHz

Output:
byte_clock_out: 81MHz
video_vsync: 25Hz
video_hsync: 80445Hz

Does this sound like what you would expect?  If so, I guess I should start looking at the image data.  When exactly do I sample video_data?  Every byte_clock_out cycle after video_vsync and video_hsync go high?
« Last Edit: June 12, 2017, 05:36:27 am by malkauns »
 

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #13 on: June 12, 2017, 09:55:32 am »
video_data is sampled based on the pixel clock, and is valid whenever video_den is high. Note that two pixels per clock are output. The pixel_clock for the default config should be 145MHz, with an MCLK of about 24.4MHz. As a quick test you could output the current pixel value to the integrated logic analyser or some other output device and confirm it decreases when you cover the camera.
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #14 on: June 12, 2017, 04:27:21 pm »
Yep, I've integrated the simple_debayer code into csi_rx_top and verified that when I shine my camera flash into the OV13850 lens, the video data (RGB at this point) is mostly 0xffffffff.  I added some quick code to count the amount of times video_den is high on each rising edge of pixel_clock_in.  This comes to roughly 102MHz.  Am I correct in assuming that the first RGB pixel of a frame occurs when pixel_clock_in, video_den, video_vsync and video_hsync are all high?  Then only read (2 pixels) at every pixel_clock_in, video_den until I have obtained 3840 pixels.  Repeat when those 4 signals are high again for each video line?
 

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #15 on: June 12, 2017, 04:43:14 pm »
Actually as per standard timing conventions with the default configs the vsync pulse will be sometime before the first line and hsync pulse sometime before the first pixel. Otherwise that all looks good.
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #16 on: June 12, 2017, 04:51:38 pm »
Ok, I'll experiment with this (probably with more questions :-DD).  Thanks for your fast reply as always and all your help so far.  Very much appreciated!!
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #17 on: June 29, 2017, 10:33:04 pm »
Hi Dave, made some progress but just need a little help in getting a valid image.  I have turned on the color bar option (which I assume is a static image, correct?) by setting register 0x5e00 to 0x80.  However, I get an image like this:



Here's a shot of the top left of that image:



I think the problem may lie in my lack of understanding of video_data_even and video_data_odd.  Am i correct in assuming that video_data_even is pixel 0, 2, 3, 4 etc. and video_data_odd is pixel 1, 3, 5, 7 etc. ?  Either that or there's something wrong in my code logic.
 

Offline cirthix

  • Contributor
  • Posts: 19
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #18 on: July 01, 2017, 11:07:47 am »
I was under the impression that the signaling levels in the mipi D-PHY were incompatibile with the inputs of most fpgas out there, including artix/zynq/kintex families.

There are hardware workarounds, but it looks like you're trying to hook it up directly to the pins.
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #19 on: July 01, 2017, 05:20:51 pm »
I was under the impression that the signaling levels in the mipi D-PHY were incompatibile with the inputs of most fpgas out there, including artix/zynq/kintex families.

There are hardware workarounds, but it looks like you're trying to hook it up directly to the pins.

I also wondered if that was the case when I started this project but everything seems to sync up.  The vsync and hsync MIPI signals are making sense in my case and I am obtaining 23 vsyncs/sec and 2160 hsyncs/sec.  Between each hsync i'm obtaining exactly 3840 pixels.  I don't think this would be the case if D-PHY signal incompatibility was an issue.  There's either something wrong in my code logic or I am not understanding things correctly.  Having said that, what are the hardware workarounds that you mention?
 

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #20 on: July 01, 2017, 06:20:53 pm »
Have been able to get completely stable performance using LVDS inputs provided signal integrity is good.

I think the problem in your case is the input "skew" settings need tweaking slightly to get the best possible performance.

Actually the way the vhdl is implemented valid video timings will likely (except vsync) will be generated even if the received input is bad.
 

Offline cirthix

  • Contributor
  • Posts: 19
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #21 on: July 01, 2017, 08:50:24 pm »
I was under the impression that the signaling levels in the mipi D-PHY were incompatibile with the inputs of most fpgas out there, including artix/zynq/kintex families.

There are hardware workarounds, but it looks like you're trying to hook it up directly to the pins.

I also wondered if that was the case when I started this project but everything seems to sync up.  The vsync and hsync MIPI signals are making sense in my case and I am obtaining 23 vsyncs/sec and 2160 hsyncs/sec.  Between each hsync i'm obtaining exactly 3840 pixels.  I don't think this would be the case if D-PHY signal incompatibility was an issue.  There's either something wrong in my code logic or I am not understanding things correctly.  Having said that, what are the hardware workarounds that you mention?

Check out xapp 585 for per-bit deskewing and xapp 894 for the d-phy stuff.

If you think your problem lies in the logic, I'd suggest using the integrated logic analyzers to look at a single line of data at a time compared to the raw input stream.

That said, I'm not sure how fast you're trying to run this, but the attachment from the fpga board to the fpc breakout board looks terrible (and that fpc breakout isnt going to help).
« Last Edit: July 01, 2017, 08:55:07 pm by cirthix »
 

Offline daveshah

  • Supporter
  • ****
  • Posts: 356
  • Country: at
    • Projects
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #22 on: July 02, 2017, 07:49:07 am »
In terms of signal integrity I would also recommend moving away from the breakout board and jumper cables if you haven't already.

The link is running at the best part of 1Gbit with low signal levels and quite tight timing requirements so you're lucky to be getting what you're getting really.
 

Offline malkauns

  • Contributor
  • Posts: 32
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #23 on: July 02, 2017, 08:29:46 am »
hmm, what would you suggest?  I'm not sure exactly how else I would be able to attach it to my Zybo board.  I'm open to buying a new dev board though if its not too expensive.
 

Offline cirthix

  • Contributor
  • Posts: 19
  • Country: us
Re: Open source CSI-2 Rx core for Xilinx FPGAs
« Reply #24 on: July 02, 2017, 09:23:33 am »
hmm, what would you suggest?  I'm not sure exactly how else I would be able to attach it to my Zybo board.  I'm open to buying a new dev board though if its not too expensive.

This should be fine
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf