The bowels of HDMI.

[hamster_nz]

If anybody is interested in using an FPGA to explore deep in the bowels of HDMI you might be interested in my new GitHub Repo.

https://github.com/hamsternz/Artix-7-HDMI-processing

It is HDMI 1080p RX and decode, audio extraction and then sending the video back out as DVI-D, but with 8 channel audio level meters that are blended into the top left corner of the video - the simplest project I could think of to show end-to-end functionality.



It is a bit rough at the moment, but should get better as I fix up a few known issues (f.e. the meters double in height for 1080i inputs).

Features
--------
Supports HDMI input formats:
  -720p@50
 - 720p@60,
 - 1080i (with a bug in the meters)
 - 1080p@50
 - 1080p@60
 and others....

Colourspaces / formats supported:
 - RGB 444   
 - YCbCr 444
 - YCbCr 422

Supported Boards
 - Digilent Nexys Video (Artix 7 200T)

Sources tested with:
 - Western Digital HD Live
 - HP Laptop
 - Cubieboard

Sinks tested with:
 - Viewsonic Monitor - HDMI input
 - AOC Monitor DVI-D input
 - Vivo TV - HDMI input

[diyaudio]

If anybody is interested in using an FPGA to explore deep in the bowels of HDMI you might be interested in my new GitHub Repo.

https://github.com/hamsternz/Artix-7-HDMI-processing

It is HDMI 1080p RX and decode, audio extraction and then sending the video back out as DVI-D, but with 8 channel audio level meters that are blended into the top left corner of the video - the simplest project I could think of to show end-to-end functionality.



It is a bit rough at the moment, but should get better as I fix up a few known issues (f.e. the meters double in height for 1080i inputs).

Features
--------
Supports HDMI input formats:
  -720p@50
 - 720p@60,
 - 1080i (with a bug in the meters)
 - 1080p@50
 - 1080p@60
 and others....

Colourspaces / formats supported:
 - RGB 444   
 - YCbCr 444
 - YCbCr 422

Supported Boards
 - Digilent Nexys Video (Artix 7 200T)

Sources tested with:
 - Western Digital HD Live
 - HP Laptop
 - Cubieboard

Sinks tested with:
 - Viewsonic Monitor - HDMI input
 - AOC Monitor DVI-D input
 - Vivo TV - HDMI input

  Nice, don't have a FPGA background, this project is a good reference if anyone want to play with something advance in the FPGA audio/video hdmi space.

do you have a YouTube video perhaps ?

[hamster_nz]

Hi DiyAudio, here is a short, supper jerky, short cellphone video....

[diyaudio]

Hi DiyAudio, here is a short, supper jerky, short cellphone video....

Thanks this looks great man, you should make a video series explaining this project, their seems to be is a serious lack of "good quality" FPGA educational videos, only fragmented knowledge. 
 

[marshallh]

Cool, that is a lot of work. You have an hdmi analyzer?

[hamster_nz]

Thanks this looks great man, you should make a video series explaining this project, their seems to be is a serious lack of "good quality" FPGA educational videos, only fragmented knowledge. 
 

I would be quite keen on doing a video, but I hate the sound of my own voice. Do I really sound like that? Sheesh!

The other problem is that the code or design isn't of itself that interesting - for example:

.... we shift one bit per cycle into these two 16-bit shift registers, and two bits per cycle into this other 32-bit shift register.

If the first 16-bit register is 0xFFFE, and the second  has 0x0282 in the lowest 8 bits, I then pick these four bits seemingly random bits out of 32-bit shift register, and then use them to set the output signals.

Code: [Select]

architecture Behavioral of extract_video_infopacket_data is
    -- For this usage, we are only interested in four bits that are all in the first
    -- 16 transfers of the 32-bit packets
    signal header_bits     : STD_LOGIC_VECTOR (15 downto 0);
    signal frame_bits      : STD_LOGIC_VECTOR (15 downto 0);
    signal subpacket0_bits : STD_LOGIC_VECTOR (31 downto 0);
    signal updated         : std_logic := '0';
begin

process(clk)
    begin
        if rising_edge(clk) then
            if adp_data_valid = '1' then
                -----------------------------------------------
                -- Move the incoming bits into a shift register
                -----------------------------------------------
                header_bits     <= adp_header_bit      & header_bits(header_bits'high downto 1);
                frame_bits      <=  adp_frame_bit      & frame_bits(frame_bits'high   downto 1);
                subpacket0_bits <= adp_subpacket0_bits & subpacket0_bits(subpacket0_bits'high downto 2);
                updated         <= '1'; 
            end if;

            ----------------------------------------------------
            -- The 0 in frame bits indicates the start of packet
            ----------------------------------------------------
            if updated = '1' and frame_bits = x"FFFE" then
                -- 82 is the type of packet, 02 is the version
                if header_bits = x"0282" then
                    case subpacket0_bits(14 downto 13) is
                        when "00"   => input_is_YCbCr <= '0'; input_is_422 <= '0';
                        when "01"   => input_is_YCbCr <= '1'; input_is_422 <= '1';
                        when "10"   => input_is_YCbCr <= '1'; input_is_422 <= '0';
                        when others => NULL;
                    end case;

                    case subpacket0_bits(27 downto 26) is
                        when "01"   => input_is_sRGB <= '1';
                        when others => input_is_sRGB <= '0';
                    end case;

                end if;
            end if;
        end if;
    end process;
end Behavioral;


The interesting bit is mapping the HDMI standard to the flow of data through the FPGA design, and in the little tidbits like the Guard Band symbols and TERC4 symbols are actually valid TMDS pixel data symbols, so you can actually move the decoding to an entirely different place than you would first have assumed... and none of that stuff actually shows up in the code.

But you wouldn't find that stuff all that interesting unless you have had some prior knowledge of the protocol.

[hamster_nz]

Cool, that is a lot of work. You have an hdmi analyzer?

No, I've got just the HDMI protocol spec (thanks Google!), an FPGA board, a few different HDMI sources/sinks and a lot of time in the simulator.

I've been leapfrogging through projects to get more and more adept
- Analog Video
  Generating basic analogue VGA signals

- TMDS / DVI-D output
  Converting those signals to DVI-D

- TMDS / DVI-D input
   Receiving and decoding high speed data.

- HDMI output
   Building valid HDMI streams that actually work

- HDMI input
   Receiving valid HDMI streams and decoding them.

Strangely enough, the most painful bit so far was EDID/DCC - which is required to get the HDMI source to send HDMI data. I've wasted so much time over something that should be so simple, due to various timing issues, checksums, lack of clear documentation and my own stupidity - I can shift bits at 1.5Gb/s, but can't get a flipping 400kHz I2C device to play ball. Grrrrr.

Oh how I now love that Saleae Logic Analyser I borrowed...

[Stebanoid]

Well done! 

I would be quite keen on doing a video, but I hate the sound of my own voice. Do I really sound like that? Sheesh!

I thought why I don't like my voice recorded:
Usually, you hear your own voice mostly through your bones which have another Frequency response, so when you hear your real voice it differs and it makes you angry. It's not a problem for other people - your voice is most likely OK for them because they don't have any preprogrammed anticipations about it.

[hamster_nz]

Oh, I've now added 'rule of thirds' guidelines too...

[diyaudio]

I would be quite keen on doing a video, but I hate the sound of my own voice. Do I really sound like that? Sheesh!

Same here, wait maybe Dave can give some help in this department. 
Maybe talking in the mirror to warm up?, I have a pretty good mic (audio technica at2020) on the desk and I feel very annoyed listing to my own voice as well

What HDMI specification doc did you use ?

[krivx]

Cool, that is a lot of work. You have an hdmi analyzer?

Have you used one? I have seen that kevtris has one and used it to validate his HDMI Nes project. It would be very nice to see an open source HDMI-compliant example, it feels like analyzers are so niche and $$$ that it hampers a lot of fun potential projects.

[Bassman59]

Oh, I've now added 'rule of thirds' guidelines too...

Shaun the Sheep! He even mucks about with those who cannot bleat!

(the movie finally opens here in the colonies this week!)

[hamster_nz]

Have you used one? I have seen that kevtris has one and used it to validate his HDMI Nes project. It would be very nice to see an open source HDMI-compliant example, it feels like analyzers are so niche and $$$ that it hampers a lot of fun potential projects.

Although could arrange access to one, I'm deliberately avoiding it. To extract video data (and to know where to replace it) all you need to know is how DVI-D works (see http://www.cs.unc.edu/~stc/FAQs/Video/dvi_spec-V1_0.pdf), and that video data is announced in advance by a block of eight CTL codes, followed with two guard band data words (that can be ignored as they don't show).

If you get a trace of the symbols the HDMI format it is pretty much self-evident - you see standard CTL (non-video) symbols, then CTL codes on Ch1 & Ch2 change for 8 cycles (Ch0 is used to transfer HSYNC and VSYNC) , followed with 1082 data symbols. If you decode the TMDS for them you get the raw pixels, with two constant values at the start of the raster.

That ignores the whole HDCP issue - however some cheap HDMI video splitters can take care of that for you - one of the channels comes out clean.

[marshallh]

Cool, that is a lot of work. You have an hdmi analyzer?

Have you used one? I have seen that kevtris has one and used it to validate his HDMI Nes project. It would be very nice to see an open source HDMI-compliant example, it feels like analyzers are so niche and $$$ that it hampers a lot of fun potential projects.

Actually I will be using his this weekend I convinced him to do a teardown a while back, there's a video if you haven't seen it.

[krivx]

Cool, that is a lot of work. You have an hdmi analyzer?

Have you used one? I have seen that kevtris has one and used it to validate his HDMI Nes project. It would be very nice to see an open source HDMI-compliant example, it feels like analyzers are so niche and $$$ that it hampers a lot of fun potential projects.

Actually I will be using his this weekend I convinced him to do a teardown a while back, there's a video if you haven't seen it.

Ha, I have seen it. I really like his videos, it's nice to see someone cover this stuff with such expertise

[hamster_nz]

Last night I extended the project to do Sobel-like edge detection:



The threshold is a little bit high at the moment so I'll fix that and get rid of the artefacts at the edges of the screen...

[davorin]

Hello Mike

I know..this topic is rather old...but I just happen to experiment with your HDMI processing design on my Arty 7 board (o;

Just a quick question....should hdmi tx generate a signal when there is no input at the rx side?
Or di I need to connect SDA/SCL on the tx side as well?


Lastly I used your Artix 7 HDMI output example and I could run it with just connecting the TMDS pairs and nothing else...

[BrianHG]

Last night I extended the project to do Sobel-like edge detection:



The threshold is a little bit high at the moment so I'll fix that and get rid of the artefacts at the edges of the screen...

The true method to get rid of edge screen line artifacts on enhancement and sharpening/blur filters.  Pay attention since altering the enhancement level isn't a good fix if you need high levels throughout...

Simplified illustration using 1 line of a 3x3 convolution matrix.  You will need to expand this on the X and Y axis as needed.  Include the Z axis if you are doing Z time domain filters like motion blur...

x1y1|x2y1|x3y1
-------------------
x1y2|x2y2|x3y2
-------------------
x1y3|x2y3|x3y3

As the video is coming in, once active video flag is enabled, for the first pixel, x1 and x2 and x3 should all be made equal to pixel 1. For the next pixel, x1 = the next pixel as x2 takes the previous x1 and x3= the previous x2.  When beginning a new scan line, all the pixels in the filter coefficients become equal meaning 0 change meaning 0 artifact.  After the end of the active video on the line, x1 should hold the previous pixel not changing.  The shift roll effect here will have the same consequence of the edge pixels off the side of the image being equal to the pixels just before erasing that edge detect artifact.

On the Y lines of video, follow the same procedure.  At the First new video line coming in, feeding y1, use a that y1 as you y2 and y3 as well.  On the next line as your y2 buffer has been now filled, use the new y1, use the buffered y2, for y3, use the existing y2.  On the next line, the new coming in y1 with the buffered y2 and the buffered y3 are all valid.  At the bottom of the video, for the line where an empty y1 is coming in since there is no more picture, use the buffered y2 data as your y1 data.  The buffered y2 will also be use as y2, y3 is still used and still gets it's data from y2...  Ect... you should have the idea.

You are essentially doing a padding of picture data where there is not picture data.  The padding data turns out to be the last pixel value data at the edge of the image.  You just need to take care in handling Y axis as the padding data is held in a moving line buffer.

BTW, great job at decoding & encoding the HDMI bitstream. 

[hamster_nz]

Just a quick question....should hdmi tx generate a signal when there is no input at the rx side?
Or di I need to connect SDA/SCL on the tx side as well?

The SDA/SCL are only needed on the source (input to the FPGA) side, so it can 'see' a valid HDMI sink.

On the output side it is DVI-D, and uses the pixel clock from the source (so no output is generated if the source is not active). It ignores the EDID info of the final device.

[davorin]

Ah okay....seems there is another problem then with the Arty 7 board....the clocking region:

Code: [Select]

[Place 30-149] Unroutable Placement! A MMCM / (BUFIO/BUFR) component pair is not placed in a routable site pair. The MMCM component can use the dedicated path between the MMCM and the (BUFIO/BUFR) if both are placed in the same clock region or if they are placed in horizontally adjacent clock regions. If this sub optimal condition is acceptable for this design, you may use the CLOCK_DEDICATED_ROUTE constraint in the .xdc file to demote this message to a WARNING. However, the use of this override is highly discouraged. These examples can be used directly in the .xdc file to override this clock rule.
< set_property CLOCK_DEDICATED_ROUTE FALSE [get_nets i_hdmi_io/i_hdmi_input/clk_pixel_x5_raw] >
And changing the TMDS input clock to 720p (13.47nsec period in constraint file and hdmi_input.vhd) gives me:

Code: [Select]

[DRC PDRC-34] MMCM_adv_ClkFrequency_div_no_dclk: The computed value 371.195 MHz (CLKIN1_PERIOD, net tmds_in_clk) for the VCO operating frequency of the MMCME2_ADV site MMCME2_ADV_X0Y1 (cell i_hdmi_io/i_hdmi_input/hdmi_MMCME2_BASE_inst) falls outside the operating range of the MMCM VCO frequency for this device (600.000 - 1200.000 MHz). The computed value is (CLKFBOUT_MULT_F * 1000 / (CLKINx_PERIOD * DIVCLK_DIVIDE)). Please run update_timing to update the MMCM settings. If that does not work, adjust either the input period CLKINx_PERIOD (13.470000), multiplication factor CLKFBOUT_MULT_F (5.000000) or the division factor DIVCLK_DIVIDE (1), in order to achieve a VCO frequency within the rated operating range for this device.
Using "BUFIO_x5_inst : BUFG" instead of "BUFIO_x5_inst : BUFIO"  the design runs through and I can program it...but the timing requirements are not set with BUFG. Remember a post on the Digilent forum that you had to use BUFIO for faster clocking.


Guess I am out of luck with my Arty 7 board (o;

[davorin]

Interesting...with the Zybo Z7 board the hdmi processing design works with feeding 1080p@60 from a Raspberry Pi...had to use BUFG as well for the x5 clock....and worst negative slack is reported with -3.874nsec.

First thought it might to do with HDMI TX/RX being in a different IO bank as only on the Nexys video board both TX/RX are on the same bank.
Maybe the Arty 7 RX side doesn't see any input signal...

[james_s]

Well done! 

I would be quite keen on doing a video, but I hate the sound of my own voice. Do I really sound like that? Sheesh!

I thought why I don't like my voice recorded:
Usually, you hear your own voice mostly through your bones which have another Frequency response, so when you hear your real voice it differs and it makes you angry. It's not a problem for other people - your voice is most likely OK for them because they don't have any preprogrammed anticipations about it.

Funny, I'm the same way, hearing my own voice doesn't make me angry, it just sounds weird and I don't like it.