Video Player?

[Wilksey]

Hi,

I have an idea of a small-ish screen with a "driver" to display something like a wedding video that I can mount to a small picture frame or something, a bit like a digital photo frame but a bit more personal.

I have seen several projects of people reading images from SD cards on Arduinos and PICs and displaying them, which seems to work well.

I have a selection of screens from eBay which work fine and I can communicate with them and I can load images onto them.

What I wanted to do was to grab a video file, use something like FFMPEG to rip the frames from the video (resized to the screen) and the audio to WAV and use a micro to display the images and sound in sync so it is effectively a movie.

Now my questions are:
1. Would I be able to read fast enough to render a smooth video? I am thinking 25fps might be too much, but what about 15 or below, I guess this is also dependant on clock speed.  I don't know if a PIC @ 16MHz or 40MHz or even 80MHz would be able to read the data from an SD card fast enough to read in and clock out the data.
2. My other thought is to use a low cost SBC such as the RPi and get one of those cheap car LCD monitors and "modify" that to mount in a frame and feed it from an older Pi's composite output.

My basic idea is a frame with a power button and a single push button, when the frame is switched on it will display a single still image, when the button is pressed it will show the "movie" and then return to the static image until the button is pressed again.

The video itself is 10 mins long, I've thought about perhaps using something like an FPGA to read the frames and clock the data out as it might be able to do it more efficiently, but I am not 100% sure how to output the audio from an FPGA, I guess either one with built in DAC or an external DAC.

Just a thought at the moment, so any suggestions welcomed!

Cheers

[NiHaoMike]

How big of a screen are you looking for?

[Wilksey]

I have a small Nokia screen which is 128x128, I also have  a 320x240 res 3 or 4" screen, if I was to use something like a RPi I could use a 5 or 7" screen.

I have a few colour screens, just depends what hardware is best suited, if I use a micro the bigger the screen the higher the res to fill it, which mans the higher file size and slower to load I should imagine.

I am open to suggestions.

[NiHaoMike]

Just get an old smartphone or tablet and set it playing video on repeat.

[Richard Crowley]

I believe I have already seen demo video on YouTube of Raspberry Pi playing video from the SD card onto an LCD screen.
Adafruit sell a rather wide range of compatible LCD screens.  Seems quite doable, IMHO.

[Wilksey]

The RPi is a last resort
I don't want to just buy a tablet or smart phone off the shelf, wheres the fun in that?! it's not a commercial project, just a bit of tinkering / fun.

I would prefer to do it with a PIC or Atmel (Arduino) if possible, but I am not sure it is up to the task speed wise.

[Richard Crowley]

The RPi is a last resort

WHY??

I would prefer to do it with a PIC or Atmel (Arduino) if possible, but I am not sure it is up to the task speed wise.

You are dreaming.  They are too slow for live video by several orders of magnitude.  Not even close.

[Wilksey]

For live video, yes, but I am talking about loading images to give the impression of a video.

RPi is quite high power so I would want to avoid using it if I could.

[Richard Crowley]

You might be able to cobble something low-power that can play animated GIFs (or somesuch) at a few FPS.
Remember that it takes more power also to operate a display capable of that kind of dynamic response.

[NiHaoMike]

Building something from scratch to play video isn't that easy. For some DIY fun, wire up a huge battery to that phone or tablet so it will run all day.

[Wilksey]

Well, I am thinking it just depends on the speed of the interface right?
That includes the speed of the display and the speed of the SD Card interface.

If I say I want 25fps that means I will need to load a new image every 40ms, 15fps every 65ms, for 5 every 200ms.
The file sizes at small res, albeit a lot of them do not amount to hundreds of megabytes, I was thinking I could perhaps use a RAM chip of a large-ish size to pre-buffer a few frames, say 5 - 10 seconds worth (perhaps less is needed in reality) to make loading quicker, and implement a kind of circular buffer where when one image is removed from the RAM a new one is loaded from SD, that way it won't matter too much if it takes slightly longer than ideal to load a new frame.

I guess a high clock speed is required, but I have also thought about the possibility of using an FPGA/CPLD to buffer the images from SD card and write to the parallel bus of the screen, but wonder if that is overkil.

[SirNick]

Assuming 8bpp (24-bit RGB) images @ 320x240, that's 1,843,200 bits per frame.  So, roughly 20Mbps at 10fps.  An AVR couldn't keep up with that even if you had perfect, 100% efficiency, such that you could clock a bit from the SD card straight to the screen with each tick.  That's not including audio.

You might be able to pull it off with a Cortex M.  Many have dedicated SD I/O and parallel TFT LCD interfaces, plus a DAC for audio.  (Maybe not terribly high quality audio, but enough.)  If you can pre-process the video into a stream that is as efficient as possible to send to the display, and make ample use of DMA, it could be doable.

[Wilksey]

SirNick,

Thanks for the input, I do have a M4 and M3 somewhere, perhaps that would be a better option as you said.

Audio quality, I am not that worried about, I am thinking now maybe use a separate MP3 playback chip and just play it at the same time as the "movie" plays, i'll have to of course look at the delays from play button pressed to track start, but that's easy enough, plus these small £3 MP3 modules have an amplifier man enough to drive a small speaker built in.

I am going to do some experiments but I wanted opinions on if I was going to be able to do it or not first off.  As the screens are not particularly "high quality" I might start of at 16 bit RGB, not that it will make that much difference in loading speed.

[SirNick]

Image quality and screen capabilities aside, if you have to send 8-bit color values to the LCD controller (or whatever) anyway, it might actually hurt performance to reduce fidelity.  Totally depends on which is faster: I/O, or whatever manipulation it takes to massage the data to the format your screen controller wants.

Just something (maybe obvious) to keep in mind.

[Wilksey]

Hi SirNick,

Yes, all good points, and I shall think about them before going further, I wouldn't change the format on the fly, I would re-jig the raw files to be 8 bit/16 bit.

I am still in thinking stage, not even design so any small pieces of information such as the one you posted is more than welcomed!

Thanks again.

[ttt]

I've had good luck with the BeagleBone Black for video applications. It's much more powerful than a RPi. I run ffmpeg directly on it without converting video into a raw format first. The two PRUs (which are MCU like co-processors) make stuff like this possible:



I use these panels: https://www.sparkfun.com/products/12584 I drive them completely by software PWM in the PRUs, no FPGA anywhere. When I say PWM, I mean that these LED panels really have no pixel buffer, you only control if the LEDs are on or off. That put this project into a different league compared to say a project using WS2801s. Granted, it took quite a bit to get it to this point  Source code for this project is posted here: https://github.com/tinic/rgb-led-panel-sign The biggest snag I ran into during this project was that most code samples you find on the web for the PRUs are doing it completely wrong and only get a fraction of the performance possible.

[ovnr]

You should look at Mike's (mike's electric stuff) videos on the iPod Nano displays. He's essentially streaming data from a SPI flash chip to a 240x240 display using a small FPGA and a PIC32.


But let's go with a micro. We'll assume you run 320xRGBx240 at 6bpp (18 bits/pixel), and that your display controller deals with a single large blob of data without issue. You're looking at 169 kB of data per image, and at 25 fps that's 4219 kBps or around 34 Mbit/s, which is a bit excessive. Dropping to 15 fps gives you 2531 kBps (~20 Mbit/s). Since you can just initialize your display controller and stream data at it (you don't have to read it into the micro and then re-send it to the display - just format it in your flash memory to fit the display controller), this greatly simplifies things.

SD cards should (it's been a few years since I had to deal with them) happily stream data out constantly, so you can just set it up (seek to the start of the video), and let it rip - temporarily connecting the SD card MISO pin to the display MOSI pin. If the cards spit out status data between each 512B block, just take the display chip select low.

(So essentially you just set things up, reconfigure some pins and maybe an analog mux or two, then issue 128 32-bit reads, disable CS on the display, read 1-4 status bytes, re-enable CS, and rinse & repeat.)

I'm sure there's a good few gotchas involved, but there's no reason you can't get it going. The SD card IO bus is spec'd to 25 MHz, so that's plenty - all you really have to do is generate a 25MHz SPI bus clock, then write some commands in between. Oh, and possibly twiddle the clock speed so that you get the correct playback speed. You'll also have to format the pictures properly before writing them to the card, and you have to write them contiguously, because this doesn't really care about filesystems (I'd probably build a binary image of the card and just dump it to the card using dd or something).


Also, I didn't touch on audio. Ignoring possible sync issues (Have fun!), I'd just dump it into some flash or SRAM and let a whole separate micro deal with it. You could surely get it out alongside the video, but it'd just be an exercise in frustration - and depending on your video length, audio should fit in a few MB of flash.


tl;dr - Some cleverness and a micro is all you need. You do not need a full-blown computer (RPi) or similar; it's just data.
Also, expect around 150 MB per minute of video, so a 2GB card is going to top out at around 13 minutes.

[Richard Crowley]

I've had good luck with the BeagleBone Black for video applications. It's much more powerful than a RPi.

But Wilksey says he is looking for something LESS POWERFUL than RPi. 
Perhaps he seeks battery operation. We don't know because he has not revealed (or maybe hasn't thought out?) his requirements.

Some of the LCD video display modules include a mini-SD card socket and will play video directly from the flash memory card.
That is probably the lowest-power practical way of meeting the requirements that Wilksey has revealed so far.

[Wilksey]

Hi Guys,

Thanks for the replies, a wealth of information to consider!

The project is not secret, I just didn't know how much information was required to disclose
The "player" will run preferably from a battery, LiPo or such with a boost converter if it needs to run @ 5v.

I was thinking micro over something like RPi / BBB due to heat dissipation and start up times.

Imagine if you will a bespoke digital photo frame that just plays a video, and that is the general idea, but I want to "hand make" it, including the micro boards, for a personal approach, another reason I want to use the RPi / <Insert a n other ARM SBC here> as a last resort, I know i could do it with a RPi etc within minutes.

I do have a few 3.2" displays from Ebay which has a SD card holder on the back, but it is broken out separately on a large dual row header, I don't think they will run directly.

For the audio, I have used FFMPEG to pull the audio stream separately, I could always use a separate audio playback module if I run into too many issues.  Synching the A/V is going to be a bit of a nightmare I expect.

[Richard Crowley]

IMHO, you are in that stage where "you don't know what you don't know" and you are in way over your head.  Perhaps after a year or two of experimenting with microcontrollers and LCD displays, you will be in a better position to evaluate the practicality of what you think you want to do.

The LCD screens I was talking about with built-in flash card readers have quite powerful microcontrollers which process the data and refresh the pixels.  They are not simply LCD screens with an additional SD socket attached.

Note that there are some rather inexpensive "digital picture frames" which will also play limited video.  It shows that it can be done. But those things were not designed by amateurs with no previous experience.  They are quite remarkable in their design and manufacture.

[Wilksey]

Well, no, i've done a lot of projects at home and at work using Microcontrollers, using various displays for things, I know how to load a bitmap on a GLCD / OLED etc for example, I have been playing with these 3.2" displays for a bit and their picture quality are OK for a few ££'s from Ebay.

A video is essentially a series of images sometimes with audio in a container format.

Rather than get the micro to attempt to rip the container apart, separate audio to video, then try and convert the compressed video into something readable at bit level is just not going to happen, so I have some circa 6000 images from my 10 minutes clip with a .wav file which contains the audio.

I am not that bothered at this stage about audio (I will be at some point in the future so I am thinking ahead), what I want to do is load each "frame" / image from an SD card onto the display using the micro controller, if I wanted these images to display every 5 seconds say, I know I can do that, but for 25 FPS I need to load an image every 40ms, for 15 FPS every 67ms, 10 FPS every 100ms, etc etc.

I have done a small experiment and I think I can safely load these at 10fps using 128x64 resolution on a micro, I have yet to experiment with larger screen resolutions and images of larger resolution.

Unfortunately "professional" items are done en masse, and for a 1 off hand made creation I wanted to do it the "creative" way.

[Richard Crowley]

Handling each frame as a separate image may SEEM simpler and easier, but it is actually a much higher bandwidth problem than dealing with a video stream. Virtually ALL video compression schemes use TEMPORAL as well as SPATIAL compression.  i.e. they have occasional full-size "key frames" and then a run of "difference" frames where only the changed parts of the image are sent.

[Wilksey]

True, they have key frames and look at changes between the key frames, but a micro controller isn't going to be able to decode even a simple MJPEG stream and convert the JPEG to a bitmap the screen can understand in a decent time scale.

I don't think it's "easier", but "necessary" to decode the frames "off-site" if you will.

Using a ARM based SOC or board such as the RPi / BBB would simplify this process as it could just load VLC or something and stream it, it does seem overkill for essentially displaying a few images though (albeit a few thousand images).  I may have to go down this route if it doesn't work out but I have not yet seen a good reason why it would not be able to.

[ttt]

I've done a video project using an MCU and WS2812s which does read preprocessed raw video data from a MicroSD card:

40x8 pixels, about 40 frames/sec. It does use a LPC812 (Cortex-M0+, 30Mhz). Performance is definitely an issue. Turns out that reading data from an SD card is fairly slow, even you use the on-chip SPI peripheral. About half the CPU time is spent just doing that. It reads packed RGB565 data from the card. In the next step it unpacks it to RGB24 and bit bangs it out to 8 GPIOs pins in parallel (hence 8 rows of pixels). I doubt that using a single SPI channel would be fast enough to drive a 320x240 LED screen at a reasonable frame rate (say >10frames/sec) especially since so much time is spent in reading from the SD card.

Have you also looked how much power the LED screen will use? It might be more than you expect To the point where using a full blown CPU might not be a problem at all... A Cortex-M3/M4 might be a possibility here though I believe they are much harder to set up.

[Wilksey]

Hi ttt,

Yes, the screen is relatively low-ish power.

Nice video, though I prefer the 8x32x32 one!

And there is a weird one with someone taking some boxes from some ones front door?!

Anywho,

Is the speed of the SPI down to the peripheral itself and the clock it uses or down to the SD card speed?  I am wondering if I can unpack it to the raw format and send it directly to the screen it will speed things up somewhat?  Not sure if it is enough though.