Creating a video encoding/streaming device

[Chris_F]

I'm playing around with the idea of starting a project to create a device to basically act as a hardware based remote desktop. It would basically take a HDMI input from the computer, compress it and then stream. I was hoping someone here would have some recommendations on what kind of parts could be used to put something like this together, e.g. a particular SoC or such. Some of the requirements as I see them:

• HDMI input capable of supporting 1920x1200@60Hz 24-bit
• Multi-core with enough processing power to handle A/V compression/decompression
• 10/100 ethernet controller
• Programmable USB 2.0 peripheral controller
• Good assortment of I/O i.e. I2C, SPI and GPIO
• Low latency!

I'm not sure if there are many options for SoCs with built in HDMI receiver. What kind of options would there be for interfacing an external HDMI receiver with 24-bit interface to a SoC? The HDMI signal will be 60Hz but I only plan on encoding a 30Hz video signal. It definitely will require a decent amount of processing power as I want to do all the codecs in software so that I can implement them myself. It will be encoding video and audio that is going out and decoding audio that is coming in. In addition to the codecs it will also need to be able to handle a web server as this is going to be web based.

[Chris_F]

Forget about 10/100M ethernet and USB 2.0 if you want 1200 60p. Consider USB3.0 or GbE.

That defeats the purpose. GbE has enough bandwidth to stream 1920x1200@60Hz uncompressed...

I want to compress the video heavily so that I can stream it over a broadband internet connection of ~5mbit/s. As for USB, that it only for transmitting audio bidirectionally as well as keyboard/mouse.

Also, I have to say that x264 is protected by multiple active patents now. Consider VP8 or theora instead.

I didn't mention H.264 but in any case this is for my own personal use, I am not trying to manufacture a product. If the MPEG-LA thinks I'm using their IP they can just sit on it.

[mariush]

HDMI is kind of a bad choice because of HDCP, the encryption which is enabled when people watch movies for example.  You're not guaranteed to have unencrypted output from the PC with digital outputs.  Same for DVI and Displayport, these also have optional encryption. There's a chance you'd get encrypted output and it would be hard to decrypt it without paying a lot of money to get a key 

You could use a FPGA to grab video frames from VGA output and send them to a hardware encoder chip and then mux the video frames and audio to a stream and send it. If you want to avoid licensing, I guess you could do motion jpeg, theora etc but you'll complicate your life.

There are such devices already if you're not aware, they're called KVM over IP .. they're expensive though.

[Chris_F]

Uncompressed 1200 60p requires 1920px*1200px*24bpp*60fps=2.99 Gbps.

If you are not making product, just use a small PC, such as an Intel NUC, with any HDMI interface card, preferably a USB 3.0 device. This will cost <$600 totally.

Any moderate high performance FPGA board will be several time the price.

I imagine that most of these capture cards will have poor latency and may not offer raw capture.

Use VLC as your streaming software. VLC has broad support of hardware decoding, and probably encoding, and it runs on any free linux, and it supports almost all popular streaming protocols.

This is a remote desktop application, not video streaming...

HDMI is kind of a bad choice because of HDCP, the encryption which is enabled when people watch movies for example.  You're not guaranteed to have unencrypted output from the PC with digital outputs.  Same for DVI and Displayport, these also have optional encryption. There's a chance you'd get encrypted output and it would be hard to decrypt it without paying a lot of money to get a key 

You could use a FPGA to grab video frames from VGA output and send them to a hardware encoder chip and then mux the video frames and audio to a stream and send it. If you want to avoid licensing, I guess you could do motion jpeg, theora etc but you'll complicate your life.

I don't care about HDCP. I am not watching Blurays on my computer remotely, there is not encrypted content that I need to be worrying about...

No device currently exists that does exactly what i want.

HDCP master key has been cracked for a long time. You can google it easily. Analog Devices manufactures HDMI decoders with HDCP decoding capability. You can simply generate a device key from the master key, and write the device key to the HDMI interface IC.

Again, I could not care less about HDCP...

[Chris_F]

Also, for what it's worth, I'm not looking for pristine visual quality. Something equivalent to baseline H.264 (or perhaps even simpler, like MPEG-1 or MPEG-2) would probably be good enough. I have no idea how much processing power something like an multi-core ARM Cortex SoC would have. I know that on my i5 desktop I have more than enough processing power on a single core to encode baseline H.264 1920x1200 at a solid 30fps using x264.

[Chris_F]

HDMI adapters offer H264 or MPEG data stream.

Latency exists, but much less than internet latency, so you can ignore it.

I want to be in control of the compression. I cannot guarantee that I will be able to do full H.264 decoding on the client as it will be web based (HTML5 + WebGL) and decoding the compressed video and then re-compressing it will only add more artifacts and latency.

There is only about 30ms latency between locations, so unless you know of a HDMI/DVI capture card that can provide raw pixel data to my application in much less than 30ms...

[Chris_F]

What do you mean "location"? Seems you are doing computer vision. In that case, you may want to process data on a FPGA board, and transfer only generated message through internet.

Computer vision? 

I want to remote into my computer at home using a web browser. The ping between my home computer and the place I would be doing the remoting is 30-40ms and I don't want to be adding 1000ms on top of that.

Maybe what I should be building is a PCIe based HDMI capture card that does nothing but fill a frame buffer with the raw pixel data that can then be read out.

[Chris_F]

30-40 ms is only average ping time. In fact, sometime ICMP packets has higher QoS priority than UDP packets. Therefore, worst case UDP latency could be a lot more.

Your buffer size has to be long enough to cover even the worst case, that is why remote desktop has latency. It is intended. Otherwise, you may encounter black screen for a short while.

In short, you can not get 30 mS latency over WAN, unless you have internet resource like google.

I don't care if if the total latency is 40ms or 250ms, as long as it is reasonable, and some consumer capture card that is going to provide you with a video stream that is delayed by 5 seconds on top of everything else is simply not going to cut it.

[Chris_F]

I already know that the network performance is adequate as I am already using an HTML5 based VNC and there is no noticeable delay. The problem is that this VNC like most uses a basic codec designed to handle typical desktop content at a moderate frame rate, and I want something that can do full video at 30fps.

[Chris_F]

Most consumer cards comes with standard USB video class driver. You can easily bypass their application software. VLC allows you to adjust buffer.

If you just want to capture screen with low latency, try this: http://www.waitwut.info/blog/2013/06/07/desktop-streaming-with-vlc/.

How does VLC help me? I need to stream this using websockets, along with other information like cursor position, cursor image, status information, keyboard events, etc...

[Chris_F]

You've been a great help.

[nctnico]

Forget about 10/100M ethernet and USB 2.0 if you want 1200 60p. Consider USB3.0 or GbE.

For hardware accelerated FFT/IFFT (used heavily in MP4 encoding), FPGA or high end DSP must be used.

No. A decent SoC developed for multimedia will have a hardware compressor capable of doing this. These have become pretty jellybean parts. Maybe even the 'new' Raspberry Pi can meet this requirement.