Low latency, high resolution media player

[mikeselectricstuff]

A customer of mine is being quite successful at selling interactive video artworks like this :
 http://cinimodstudio.com/project/flutter/ (which I designed, using playback from NAND flash via a small FPGA)
and this
 http://cinimodstudio.com/project/baby-flutter/ (based on a beagleboard)
and also large-scale stuff like this
 http://cinimodstudio.com/project/deep-blue-interactive-aquarium/ (uses very high-end PCs for realtime rendering)

We're looking at scaling up the smaller units up to HD type resolutions, and are drawing a blank on suitable hardware platforms. Most video playback slutions are unsuitable as we need minimal latency displaying randomly selected frames at 60fps, so anything with pipeline delays or keyframe+difference compression isn't responsive enough, and high image quality is important, so can't use much compression.
In addition to fast interaction, another requirement is fanless operation and reasonable power efficency, so we can;t just throw a turbo-nutter PC at it.

My thinking is that instead of looking at mid-high end processors, a better approach is probably to just store uncompressed images and fire them out to HDMI as required with an FPGA.
Either a few pages of frame buffer plus a fast SSD, or enough RAM to cache a substantial amount of content (a few Gbytes) and a slower mass storage solution.
Or maybe use an SSD with minmal buffer and rely on the SSD buffer/cache for frame refresh.
 It's not a problem if it takes a while (a minute or two) at startup to load up the RAM.
The actual processing will be very low-effort - taking inputs from sensors and deciding which pre-rendered frames to display when.
the content would be written once and rarely changed, so the playback system would not need to worry about any writing, and teh media can be formatted to make it easy - e.g. absolute block addressing.

The volumes could get into the low hundreds, so a custom solution may be viable - I've had a quick look at various boards but can't immediately  find anything with a lot of RAM (or DIMM socket).

The hardware I'd envsage would look something like:
FPGA (guessing Spartan-6 sort of level), DIMM socket(s) for expandability, SATA or high speed SD card. Low-speed interface (SPI/UART) to a seperate host processor (Raspberry Pi or something comparable) and HDMI/DVI out.
Passive cooling - the board could be arranged to suit cooling via a large heatsink and/or heatpipe, but no fan - it must be silent.
Looking at RAM bandwidth, we're nowhere near bleediong-edge. at about 250MBytes/sec (1024x1280x24bppx60fps).
A build cost in 100x of up to, say £300-ish would be OK.

Although it could in principle drive LVDS panels direct, a concern is supporting different connectors etc. on different panels. We're currently thinking that HDMI may be a more useful output as it gives a lot of flexibility, and allows for driving projctors etc.

I'd be interested in opinions on :

Any off-the-shelf solution that might come close (or be adaptable)

As a relative FPGA Newbie, how hard is it likely to be to get  an FPGA+memory+DDR+HDMI solution working?

I know most FPGA vendors have some solutions to help with memory, HDMI etc., but how easy are these to use in practice? I have managed to get HDMI in/out working on an
Atlys board with ready-rolled modules, but only at medium resolution.

What might it cost to contract out the hardware and FPGA design to do something like this. I know enough to be able to  produce a very well-defined detailed spec, and we can leave all the stuff like filesystem management to the host processor that we would be in charge of implementing - the custom part would simply be a display engine that displays specified blocks on command.


 

[nctnico]

You don't have to use many keyframes. A lot depends on the encoding. On my PC I can skip forward/backward instantly through most x.264 videos. I'd look into a solution around an iMX5x from Freescale which has hardware decoding to see how much latency it has. I tinkered a bit with an iMX51 board which has an HDMI output and it could play video nicely.

edit: IMHO the iMX53 series from Freescale looks powerful enough for what you want to do (I've used the iMX51 in a design). The iMX series does most of the heavy lifting and display handling in dedicated hardware so the CPU and memory bus are free for other processing tasks. The DM3730 on the Beagle board does almost everything in software (its basically an ARM core and a DSP core) which puts constraints on memory bandwidth. I'd investigate going the SoC route before committing to a dedicated FPGA device. Many years ago I designed a data acquisition board which uses standard PC DDR memory (and supports different memory configurations) together with a Spartan 3 FPGA. Getting the 64 bit wide DDR200 memory up&running did take some effort. OTOH 1.5GB/s sustained throuhgput is pretty neat :-)

Besides that it may be interesting to look into motion JPEG. This is basically a string of independant JPEG images.

[Legit-Design]

About going fanless: "turbo-nutter PC" without fans http://www.quietpc.com/sys-a470
http://www.quietpc.com/sys-a480
No idea if it's going to cost too much for driving 100 displays, but definitely no need to have 100 actual pc:s driving the displays. Also no need for fully custom hardware and design costs.
Reasonable power efficiency depends on compared to what.

PC already has everything required right out of the box:
RAM bandwidth
Sata ports for fast ssd
Dual/triple/quad? HDMI/DVI with additional graphics cards.

PC should be able to drive multiple displays, or how critical is the latency for different images for separate displays?

Should be a lot easier to have it displaying 60fps pictures with low latency, if that's the final requirement. More or less plug and play. I don't think it actually needs hardly any graphics grunt if pictures is all that's required.

Could even boot from memory stick and load images to huge ram, up to 32GB with normal consumer grade MB.

I have no experience about those particular systems, but I see no reason why they wouldn't be reliable and all that.

I know hobbyists have ran their custom built fanless silent pc:s for years already. Using heatpipes and custom machined copper blocks into big heatsinks.
How critical is the fanless requirement? Modern good quality fans can be near silent in most situations. Scythe and Noctua make extremely silent fans.  With big heatsinks fans don't need high rpm to sustain adequate cooling. I guess there are other reasons for going fanless, like maintenance or not having to clean the dust filters.

Few years ago I had cabin heater from old car as water cooling for processor only. Without any fan for the radiator, rest of the pc still had fans. Cabin heater radiator was around 2x120mm fan sized. It ran fanless at stock speeds and voltages, overvolting for more megahurtz made it unreliable, so it needed more cooling.

[amyk]

Could procedural generation techniques be used to cut down (static) storage requirements?

[grumpydoc]

How many frames do you want to store - uncompressed full HD video sucks up quite a bit of storage, at 60fps you're looking at just under 300MB per second of video so even 32G of RAM is only going to hold about 90 seconds.

If you compress the video then you either won't need so much RAM or will be able to up the duration (or a bit of both) - I suspect if you want to do it with an FPGA your options are jpeg for each frame, motion jpeg or I-Frame only H.264

That said an alternative might be a SoC based system attached to each display which is fast enough and then you can attack the problem with software tools (I presume you have played with ffmpeg) which might make it easier - as you say achieving full HD at 60fps is difficult but if you can drop to 30fps something like a cubieboard2 might be OK (if you have the budget to put one behind each screen).

It has 4GB NAND flash - using I-Frame only H.264 (say you might get away with 10MB/sec depending on source material) you could get up to 5 minutes of video with no additional storage. Seeking on an I-Frame only stream is very fast so that would probably be OK for interactivity.

If you are happy to design a system then the Allwinner A23 or A31 both claim to be capable of 1080p@60fps but I'm not aware of a pre-built SBC which uses either.

Edit: Actually the frames rates quoted above are for encode, now that I check again - decode for the A20 (cubieboard2) is rated at 3840x1080@30fps so 1920x1080@60fps should be possible, so a cubieboard2 attached to each display should be able to do what you want and at less than £60 a pop well within your budget.

[Stonent]

Will this be true HDMI encryption with audio or just straight raw DVI at 1080p60?

[mikeselectricstuff]

Will this be true HDMI encryption with audio or just straight raw DVI at 1080p60?

Just raw video - pre-rendered framed from 3D models. No audio

[ConKbot]

If going with a PC based solution, look into daisy chaining displayport monitors with displayport 1.2/multistreaming  a lot of video cards support upto 6 monitors, and not all super high end gaming cards either. 

You'd have to check into other restrictions first, but if you can run 6 monitors per card with a triple or quad card system, 18-24 displays per system may be feasible, $800-$1k for cards per system and ~$800 for the rest of the PC.  (just SWAG on those numbers though)

[mikeselectricstuff]

I really don't think a PC based solution is the way to go - you're burning a lot of power for something taht doesn't actually need to do much processing. I suppose it might be worth looking at extreme underclocking, but I suspect anything that has high enough memory and display bandwidth will tend to come with a lot of additional high-speed baggage.
the final product wants to look like a framed picture, so needs to be a thin assembly mounted  on the back of an LCD panel. We can use a large aluminium  backplate for passive cooling, but if it's just an FPGA and some RAM I can't seeing it needing much.

[daveshah]

The dual/quad core versions (and possibly the single core version) of the Freescale I.MX6 processor can address up to 4Gbyte of DDR3, which might be useful. They also have built-in HDMI and two LVDS outputs.

[Marco]

I wouldn't discount H.264, purely for image coding it's still close to the state of the art.

There are ARM boards which can decode and outpu 1080p60, but often very high bandwidth videos (such as intra-only) can make decoders stumble so you'd have to test whether they can handle it. With hardware which can handle it you just need to cobble together a transcoder to feed the hardware decoder a single continuous video, most of the code for that is already available open source.

Still seems a hell of a lot less work than a FPGA solution and with a hell of a lot better compression than you could manage in it.

[orion242]

These guys make industrial PCs, some are nothing more than a panel mountable touchscreen.  No fans, low power, and as much horsepower as  you want.  Prices from $500 to the sky depending on what you want.  Used them a few times in nasty environments and was pretty impressed.

http://www.advantech.com/default.aspx

[grumpydoc]

but often very high bandwidth videos (such as intra-only) can make decoders stumble so you'd have to test whether they can handle it.

Lightly compressed video is usually easier to handle from the decoder point of view, because there's less work to decode it. Where I have seen problems is that the software has been sloppily written with fixed sized buffers which then overflow with the larger input data (if you think about it the output data is the same size and same bitrate however highly or lightly compressed the input). Slow storage systems which can't deliver the data fast enough to the decoder can also be an issue.

I still think that something like a cubieboard2 per display is the way to go.

I-frame only H.264 compresses better than straight JPEG or motion JPEG as far as I know and pretty much by definition it will take less than 1/60th of a second to decode a frame and start showing it on the screen.

[mikeselectricstuff]

I-frame only H.264 compresses better than straight JPEG or motion JPEG as far as I know and pretty much by definition it will take less than 1/60th of a second to decode a frame and start showing it on the screen.

By definition you can decode at 60fps, but that does not guarantee low latency, as it could be pipelined, or interleaved over multiple parallel hardware accelerators, so the delay between specifying a particular frame and it appearing could be longer - I don't know enough about how these things work to know  if this is an issue in practice.

[grumpydoc]

By definition you can decode at 60fps, but that does not guarantee low latency, as it could be pipelined, or interleaved over multiple parallel hardware accelerators, so the delay between specifying a particular frame and it appearing could be longer - I don't know enough about how these things work to know  if this is an issue in practice.

I know what you mean but don't think it would be an issue - most of the latency comes from large GOP sizes and the need to process several frames to get at the one you're interested in - and you can get rid of all that with an I-frame only stream.

You haven't said how much video you want to store but at 60fps full HD a GB is going to last you three seconds, four if you want 1024x1280 rather than 1080x1920.

And cubieboards aren't exactly expensive for you to try it out.

Of course I'm more a software guy (at least that's where I have formal training) so would tend to prefer a solution which doesn't involve me designing building hardware from scratch - I suspect your perspective differs somewhat

[denizcan]

I don't think he needs any compression. The computers are not good for realtime stuff.. At least if you are running windows.. Compression would add some delay, and make thinks complicated. A raw 1920 x 1080 x 32 bit frame is just 8MB. It is really easy and cheap to put enough RAM for 1000s of frames.. This will also make the images super sharp.. No FPGA is needed, even a raspberry pi might be platform of proof of concept.. Loading RAW images from SD card to the RAM, and playing them wouldn't need much effort.. The only problem is the address range a 32 bit MCU supports, however, even extending it does not require an FPGA.. A few logic glue, or maybe CPLD is better..

[nctnico]

You are forgetting that you need a sustained throughput of 360MB/s and at least some form of double buffering in the frame buffer to avoid flicker. I doubt the Raspberry pi can do either. A 32 bit DDR200 memory system has a peak throughput of 800MB/s. Because the frame buffer is usually in the main memory you already loose half of the memory bandwidth. If you need to move 360MB/s from the main memory to another place in the main memory and display a frame at the same time you'd need over 1080MB/s.

[grumpydoc]

You are forgetting that you need a sustained throughput of 360MB/s and at least some form of double buffering in the frame buffer to avoid flicker. I doubt the Raspberry pi can do either. A 32 bit DDR200 memory system has a peak throughput of 800MB/s. Because the frame buffer is usually in the main memory you already loose half of the memory bandwidth. If you need to move 360MB/s from the main memory to another place in the main memory and display a frame at the same time you'd need over 1080MB/s.

I was going to make the same point but you beat me to it - it's the sustained rate that is the problem. You can't pull uncompressed frames off of a SD card fast enough. Even cards which exceed class 10 would only be doing a few frames per second  which is some way off Mike's spec.

The allwinner A20 has a DDR3 interface which should be fast enough - I think the Cubieboard2 has been benchmarked at 2GB/s - I'm sure that there are other SoC's which would also work well enough.

It is really easy and cheap to put enough RAM for 1000s of frames

Really?

OK, so it would be 6G (a 1920x1080 frame is 6MB rather than 8 ) for 1000 frames. That, let's remind ourselves, is 16 seconds of video at 60fps. Now the installation that Mike describes has multiple screens - not sure how many but let's say 20. Unless you can send 360MB/s to all nodes simultaneously this means local RAM so you now need 120GB in the whole system for 16 seconds worth of video per screen.

Easy, perhaps but not cheap. If you need more than 16 seconds of video available for playback the RAM required starts to go up. 30 seconds and you're at 12G, 60 and you need 24G - that's now not "easy" either.

JPEG compression can achieve 20:1 at reasonable quality which means a lot less RAM and you can start to consider getting frames off an SD card although you need one which can do better than 18MB/s. I think I-Frame only H.264 should get better ratios and both should be do-able in an FPGA if that's what Mike wants. Both should also suit a SoC solution. As I said previously if I were building this I'd at least try a Cubieboard2 per display. I don't think a Pi has the horsepower or the RAM really although Mike hasn't said how many seconds video he wants available.

[nctnico]

The SoC on the Raspberry pi is actually intended for video-over-IP setop boxes. Therefore it has a hefty hardware video decompression unit. The downside is that Broadcom doesn't provide any specifications or support unless you buy >6 digit quantities.

[grumpydoc]

The SoC on the Raspberry pi is actually intended for video-over-IP setop boxes. Therefore it has a hefty hardware video decompression unit.

I'm aware of that - however it's spec is 1080p @ 30fps, Mike asked for 60fps although subsequently hinted that he was only after 1280x1024. The Mail GPU in the A20 is speced at 3840x1080@30fps (for 3D) so should be able to do 1920x1080 at 60fps with no problems.

[mikeselectricstuff]

To avoid any confusion, there will not be multiple screens (or if there are, they will each have their own  controller and locally stored content).

Resolution vs.framerate is flexible, but 1024x1280@30fps would be an absolute minimum, but the more options the better. We'd like to drive LVDS panels direct, so this would probably need 60fps refresh from video RAM, though 30fps compression with buffering should work.
Latancy is the key thing we need to keep under tight control.

And there are quite a few sysem-on-module products  based on the Freescale i.MX6 processor that look at least worth spending time investigating, and if they will do the playback they also open the possibility for generative/rendered content.
It looks like it has all we need - SATA, reasonable RAM, HDMI, LVDS (even MIPI for tablet-scale displays) and viceo decompression hardware. With a ready-made module,  we can add a custom carrier PCB with just the stuff we need without getting into many-layer PCB, length matching DDR3 & all the other voodoo stuff.

We probably need to find a Linux (or bare metal0 low-level person to delve deeply enough to evaluate performance, as standard drivers may not do quite what we want, or do it fast enough.  (any offers please PM me - there is a development budget)
 The other aspect is that once the low-level video stuff is done, it needs to be programmable for the behavioural & content aspects by non-elite level, non hardware-oriented programmers.

[grumpydoc]

To avoid any confusion, there will not be multiple screens (or if there are, they will each have their own  controller and locally stored content).

Ah, all of your examples were multi-screen so I assumed that the current project would be also.

We probably need to find a Linux (or bare metal0 low-level person to delve deeply enough to evaluate performance, as standard drivers may not do quite what we want, or do it fast enough.  (any offers please PM me - there is a development budget)

YHM

[DutchGert]

What about using Qseven modules and bring out just the interfaces u need?

Low-power, powerfull, easy to interface with and fast development.

http://www.qseven-standard.org/

[grumpydoc]

Resolution vs.framerate is flexible, but 1024x1280@30fps would be an absolute minimum, but the more options the better. We'd like to drive LVDS panels direct, so this would probably need 60fps refresh from video RAM, though 30fps compression with buffering should work.

I see now where the "60fps" requirement comes from - more that you want to drive the panel at 60Hz rathar than decode video at 60fps.

You are forgetting that you need a sustained throughput of 360MB/s and at least some form of double buffering in the frame buffer to avoid flicker. I doubt the Raspberry pi can do either. A 32 bit DDR200 memory system has a peak throughput of 800MB/s. Because the frame buffer is usually in the main memory you already loose half of the memory bandwidth. If you need to move 360MB/s from the main memory to another place in the main memory and display a frame at the same time you'd need over 1080MB/s.

Re-reading your comment I don't think it's that bad.

For one thing everyone (including me) has been thinking in terms of the amount of storage for uncompressed RGB frames  which is not the most sensible way to do it - you really want to avoid colour space converions in software if you can so you'd keep the decoded frame as YCbCr - owing to chroma subsampling this should be half the size at 3MB per frame.

You also don't copy the data around in RAM - at least not if you can help it - just tell the GPU where the start is and Mike's already said that decoding and caching frames as part of the system start up is acceptable so you only really need to sustain 180MB/s out of RAM for the display.

It looks like it has all we need - SATA, reasonable RAM, HDMI, LVDS (even MIPI for tablet-scale displays) and viceo decompression hardware. With a ready-made module,  we can add a custom carrier PCB with just the stuff we need without getting into many-layer PCB, length matching DDR3 & all the other voodoo stuff.

Cubieboard2 will do that for you. Which Freescale boards(s) have you looked at?

We probably need to find a Linux (or bare metal0 low-level person to delve deeply enough to evaluate performance, as standard drivers may not do quite what we want, or do it fast enough.  (any offers please PM me - there is a development budget)

You probably don't need to get too close to the metal for the video side - SDL+ffmpeg+appropriate hardware acceleration library should do you. Pretty certain that you can do hardware accelerated ffmpeg on the Freescale. You might need a driver for a touchscreen if that's how you want to interact with it.

I presume whan you mean low latency you want the system to respond quickly to user interaction (ideally I'd guess within one frame)

Probably H.264 video with a very short GOP (either Iframe only or perhaps a GOP length of 2 or 3 with no B frames) will do for the storage - I suspect you will be able to put your video on an SD card and decode on the fly for most or all of the platforms that have been discussed.

The other aspect is that once the low-level video stuff is done, it needs to be programmable for the behavioural & content aspects by non-elite level, non hardware-oriented programmers.

Non-programmers I'd have thought.

[denizcan]

You guys always have to be against something..

You are forgetting that you need a sustained throughput of 360MB/s and at least some form of double buffering in the frame buffer to avoid flicker.

Did I said "proof of concept"? Let say it is not 60fps, but 30fps, ok we could not manage 1920x1080, make it 1280x720.. What we loose? 50$ of a board. Mike spends much more money to things to take it apart.. At most it may lay around to play with.. But we have tested our idea.. Did I said quickly? In a day, one can create a program that plays frames.. The rest is the sensor processing..

OK, so it would be 6G (a 1920x1080 frame is 6MB rather than 8 ) for 1000 frames. That, let's remind ourselves, is 16 seconds of video at 60fps. Now the installation that Mike describes has multiple screens - not sure how many but let's say 20. Unless you can send 360MB/s to all nodes simultaneously this means local RAM so you now need 120GB in the whole system for 16 seconds worth of video per screen.

I calculated raw 32 bits, you may use 24 bits, YUV etc.. I think you also missed what Mike aimed.. He wants to solve the problem at most 300$ per screen.. So, one board per screen.. $50 board passes this, right?