Author Topic: Looking for high speed USB memory stick or SDHC reading solution  (Read 21691 times)

0 Members and 1 Guest are viewing this topic.

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 8515
  • Country: us
    • SiliconValleyGarage
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #25 on: January 27, 2013, 08:58:52 pm »
I would go with the vinculum as well. It can even handle the filesystem. Look at vinculum 2 in special.
The VNC2 would be exactly what I need IF it were high speed, but it isn't.
They also have a new host that is high speed.

hang on ...

FT120. take a look at that. and it has a parallel data/address bus. so you should be able to plunk that on an fpga and crank away.
« Last Edit: January 27, 2013, 09:01:30 pm by free_electron »
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #26 on: January 27, 2013, 09:00:02 pm »
Quote
Just to get things clear: you want to use a USB stick OR an SD-ish card to store a file and dump it into an FPGA?
Yes - the card will be written on a PC, I just need to read it and throw out the data in the right format at a fixed rate. What I ultimately need to do is spit it out of multiple UARTS as parallel streams (Yes this is for driving lots of LEDs!)
My current solution is a PIC24FJ256GA006, outputting on its own 4 UARTs and also feeding a fast SPI stream  to a second PIC24 which then outputs the other 4 streams on its UARTs. This does the job fine for the last project (4600 LEDs) but the next one will be over twice the size, probably needing about 20 UART streams, so a <insert fast filesystem here> feeding an FPGA to do the UARTs using internal RAM to do the de-interleaving is  probably going to be a better solution.

Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline firewalker

  • Super Contributor
  • ***
  • Posts: 2450
  • Country: gr
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #27 on: January 27, 2013, 09:03:48 pm »
What if you split the file and use 2xcurent set up. Will it be difficult to sync or something?

Any infos with the led cube?

Alexander.
« Last Edit: January 27, 2013, 09:05:58 pm by firewalker »
Become a realist, stay a dreamer.

 

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #28 on: January 27, 2013, 09:06:52 pm »
Any infos with the led cube?

Alexander.
Awaiting documentation video to be done - if you're in London you can go see it at http://www.thecrystal.org/_html/visit-the-crystal.html
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 8515
  • Country: us
    • SiliconValleyGarage
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #29 on: January 27, 2013, 09:09:45 pm »
what if you do the heavy lifting on the PC?

Write the file in such a format that 1st byte is 1 stream ,second byte is second stream , 3 byte is third stream and so on.
that way you only need to open 1 file and do a cyclic step through it. hit the end and begin again.  that eliminates tons of overhead.

4600 led's each requireing what ? 1 byte for intensity control. let's say you do 50 refreshes a second.
thats 4600x50 bytes per second or 1.84 megabit per second you need to pump. the vinculum2 can do 12 megabit per second.

use a simple FPGA to do the byte demultiplexing and throw it in custom shift registers that spit out the streams. you can create all the serial ports you want in the fpga. since you do not use handshaking nor need to read back you can simply build an 8 bit shifter under control of a master clock generator. when a parallel load is done into the shift register it begins spitting out data at the tick of the master clokc. if you really need rs232 its a matter of making a 11 bit shift register holding the 2 start bit , 8 databits and the stopbit.

minimalistic code ont he vinculum opens a fixed filename and begins streaming it through the parallel fifo bus ( no need to resreialize . the vinculum spits out bytes in parallel on a clock base. you can have the fpga deliver this clock to the vinculum.
basically the vinculum opens the file and waites for a 'command; form the fpga'. the fpga says 'next byte' and the vinculum puts it on its outgoing databus. when last byte is reached you handshake through another signal adn the whole thing begins again.
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline marshallh

  • Supporter
  • ****
  • Posts: 1462
  • Country: us
    • retroactive
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #30 on: January 27, 2013, 09:11:56 pm »
If you require precise timing, complete dependence on data being available, you will want to buffer the data in some sdram. How much data are we talking about?
32mb or 64mb sdram is available on the Terasic DE0 devboard (cheap) you can buffer locally in some blockram FIFOs to get around the sdram refreshes.
Verilog tips
BGA soldering intro

11:37 <@ktemkin> c4757p: marshall has transcended communications media
11:37 <@ktemkin> He speaks protocols directly.
 

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #31 on: January 27, 2013, 09:36:53 pm »
If you require precise timing, complete dependence on data being available, you will want to buffer the data in some sdram. How much data are we talking about?
32mb or 64mb sdram is available on the Terasic DE0 devboard (cheap) you can buffer locally in some blockram FIFOs to get around the sdram refreshes.

Probably only a few tens of K - I need a consistent framerate with frames of the order of 10Kbytes, at maybe up to 50fps. I typically need to put a whole frame in RAM as it will be sliced out over multiple UARTs, and double-buffer it. Internal RAM on an FPGA should be plenty
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #32 on: January 27, 2013, 09:41:45 pm »
what if you do the heavy lifting on the PC?

Write the file in such a format that 1st byte is 1 stream ,second byte is second stream , 3 byte is third stream and so on.
that way you only need to open 1 file and do a cyclic step through it. hit the end and begin again.  that eliminates tons of overhead.

4600 led's each requireing what ? 1 byte for intensity control. let's say you do 50 refreshes a second.
thats 4600x50 bytes per second or 1.84 megabit per second you need to pump. the vinculum2 can do 12 megabit per second.

use a simple FPGA to do the byte demultiplexing and throw it in custom shift registers that spit out the streams. you can create all the serial ports you want in the fpga. since you do not use handshaking nor need to read back you can simply build an 8 bit shifter under control of a master clock generator. when a parallel load is done into the shift register it begins spitting out data at the tick of the master clokc. if you really need rs232 its a matter of making a 11 bit shift register holding the 2 start bit , 8 databits and the stopbit.

minimalistic code ont he vinculum opens a fixed filename and begins streaming it through the parallel fifo bus ( no need to resreialize . the vinculum spits out bytes in parallel on a clock base. you can have the fpga deliver this clock to the vinculum.
basically the vinculum opens the file and waites for a 'command; form the fpga'. the fpga says 'next byte' and the vinculum puts it on its outgoing databus. when last byte is reached you handshake through another signal adn the whole thing begins again.
I need to be able to tell a customer to take the video, shove it through Virtualdub with the right settings to produce an uncompressed AVI and put it on the memory device, so funky formats are not an option.
I doubt the Vinculum will get anywhere near 12mbits sustained transfer rate - that's just the raw USB rate. However a High speed host should have enough spare bandwidth to be able to waste a lot and still be fast enough.
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline Psi

  • Super Contributor
  • ***
  • Posts: 9889
  • Country: nz
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #33 on: January 27, 2013, 09:47:32 pm »
As nice as Virtualdub is, it's quite old now and cant read newer video files, like mp4 flv mkv etc..
Greek letter 'Psi' (not Pounds per Square Inch)
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26755
  • Country: nl
    • NCT Developments
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #34 on: January 27, 2013, 09:55:32 pm »
If you require precise timing, complete dependence on data being available, you will want to buffer the data in some sdram. How much data are we talking about?
32mb or 64mb sdram is available on the Terasic DE0 devboard (cheap) you can buffer locally in some blockram FIFOs to get around the sdram refreshes.

Probably only a few tens of K - I need a consistent framerate with frames of the order of 10Kbytes, at maybe up to 50fps. I typically need to put a whole frame in RAM as it will be sliced out over multiple UARTs, and double-buffer it. Internal RAM on an FPGA should be plenty
If you use a fast (ARM) controller which has a data/address bus and seperate UARTs (with FIFO) you might be able to pull this off without an FPGA. You might underestimate the complexity of the FPGA design because you may want / have to construct your own UART. The UARTs available from Opencores are not the best to put it mildly.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline marshallh

  • Supporter
  • ****
  • Posts: 1462
  • Country: us
    • retroactive
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #35 on: January 27, 2013, 10:29:05 pm »
The UARTs available from Opencores are not the best to put it mildly.

Open sores, as I call it. And yes they are total crap.
Verilog tips
BGA soldering intro

11:37 <@ktemkin> c4757p: marshall has transcended communications media
11:37 <@ktemkin> He speaks protocols directly.
 

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #36 on: January 27, 2013, 10:36:37 pm »
As nice as Virtualdub is, it's quite old now and cant read newer video files, like mp4 flv mkv etc..
I like it because it does a simple job very well with no unnecessary bells and whistles.
It adoesn't complain when I want to output bizarre uncompressed video formats like 800x1 pixel monochrome, 16 bit 5:6:5 colour or (as I'm doing right now) 4x1 pixels RGB with a mono 48K audio.
I can also hack together a filter for things like error diffusion dithering for smoother 5:6:5 displays by modding an example filter without needing to know too much about the plugin architecture.
I can just tell a customer to go download it (no installation needed) , set a few settings & it just works.
Maybe there are other options out there but if it ain't broke....
 
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #37 on: January 27, 2013, 10:37:34 pm »
The UARTs available from Opencores are not the best to put it mildly.

Open sores, as I call it. And yes they are total crap.
A transmit UART is so trivial to implement I wouldn't even think to look at using a ready-made one. 
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #38 on: January 27, 2013, 10:41:59 pm »
If you require precise timing, complete dependence on data being available, you will want to buffer the data in some sdram. How much data are we talking about?
32mb or 64mb sdram is available on the Terasic DE0 devboard (cheap) you can buffer locally in some blockram FIFOs to get around the sdram refreshes.

Probably only a few tens of K - I need a consistent framerate with frames of the order of 10Kbytes, at maybe up to 50fps. I typically need to put a whole frame in RAM as it will be sliced out over multiple UARTs, and double-buffer it. Internal RAM on an FPGA should be plenty
If you use a fast (ARM) controller which has a data/address bus and seperate UARTs (with FIFO) you might be able to pull this off without an FPGA. You might underestimate the complexity of the FPGA design because you may want / have to construct your own UART. The UARTs available from Opencores are not the best to put it mildly.
Yes I can probably do it all in software, except possibly for the "20 UARTS" part. Although it can be bit-bashed even at quite high rates (I've done 8x250kbaud on an AVR in the past)  the amount of bit-shuffling you need to do can get excessive and easier to do in an FPGA, especially if your filesystem is tight on bandwidth - you don't want to slow it down with a high interrupt load.
The timing stuff is pretty easy as long as the worst-case latency plus loading time is less than one frame period. Background task throwing out a frame on a timer int with a two buffer swap, and a foreground task filling the buffer. Gets messier if you have unpredictable latency (e.g. cluster boundaries), but even then it's usually just a case of adding more buffers to make a FIFO.
« Last Edit: January 27, 2013, 10:45:22 pm by mikeselectricstuff »
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline sca

  • Regular Contributor
  • *
  • Posts: 68
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #39 on: January 27, 2013, 11:21:23 pm »

Was this SD in SPI mode or full 4-bit? Documentation on the latter appears scarce as you need to join the SD card club to get it.
This may be a viable approach, even just  doing it a sector at a time.

Not sure if it meets any of the other requirements, but, IIRC, the LPC2300 range has an MCI port with 4 bit SD, and versions of the elm-ChaN Fat-FS stack support it.

sca
 

Offline andersm

  • Super Contributor
  • ***
  • Posts: 1198
  • Country: fi
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #40 on: January 27, 2013, 11:28:15 pm »
They also have a new host that is high speed.

hang on ...

FT120. take a look at that. and it has a parallel data/address bus.
Nope, not that one either. The FT313H is the one you want.

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #41 on: January 27, 2013, 11:30:16 pm »
They also have a new host that is high speed.

hang on ...

FT120. take a look at that. and it has a parallel data/address bus.
Nope, not that one either. The FT313H is the one you want.
That's just a host peripheral - need to add an MCU plus USB stack plus filesystem
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26755
  • Country: nl
    • NCT Developments
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #42 on: January 27, 2013, 11:35:25 pm »
If you require precise timing, complete dependence on data being available, you will want to buffer the data in some sdram. How much data are we talking about?
32mb or 64mb sdram is available on the Terasic DE0 devboard (cheap) you can buffer locally in some blockram FIFOs to get around the sdram refreshes.

Probably only a few tens of K - I need a consistent framerate with frames of the order of 10Kbytes, at maybe up to 50fps. I typically need to put a whole frame in RAM as it will be sliced out over multiple UARTs, and double-buffer it. Internal RAM on an FPGA should be plenty
If you use a fast (ARM) controller which has a data/address bus and seperate UARTs (with FIFO) you might be able to pull this off without an FPGA. You might underestimate the complexity of the FPGA design because you may want / have to construct your own UART. The UARTs available from Opencores are not the best to put it mildly.
Yes I can probably do it all in software, except possibly for the "20 UARTS" part. Although it can be bit-bashed even at quite high rates (I've done 8x250kbaud on an AVR in the past)  the amount of bit-shuffling you need to do can get excessive and easier to do in an FPGA, especially if your filesystem is tight on bandwidth - you don't want to slow it down with a high interrupt load.
The timing stuff is pretty easy as long as the worst-case latency plus loading time is less than one frame period. Background task throwing out a frame on a timer int with a two buffer swap, and a foreground task filling the buffer. Gets messier if you have unpredictable latency (e.g. cluster boundaries), but even then it's usually just a case of adding more buffers to make a FIFO.
Serving 20 UARTs isn't a problem if they have to transfer the same amount of data. You only need to use one interrupt from the last UART you serve. If the last UART's FIFO is half emtpy (*), you can be sure you can dump half a FIFO size worth of data into the UART. If the data bus is wide enough you might be able to write 2 or 4 bytes to the UARTs in one cycle.

(*) Choosing the right low water mark takes some tuning.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #43 on: January 27, 2013, 11:56:02 pm »
Quote
Serving 20 UARTs isn't a problem if they have to transfer the same amount of data. You only need to use one interrupt from the last UART you serve. If the last UART's FIFO is half emtpy (*), you can be sure you can dump half a FIFO size worth of data into the UART. If the data bus is wide enough you might be able to write 2 or 4 bytes to the UARTs in one cycle.

(*) Choosing the right low water mark takes some tuning.
The problem is not so much serving 20 UARTs, but finding an MCU that has 20 UARTS to service..!
I also sometimes need to add a little inter-byte padding e.g. 3.5 stopbits due to constraints at the receive end (typically a couple of hundred PIC10F322s doing a soft rx UART at 333kbaud) ,  this is doable using a timer int instead of UART ints but  precludes the use of UART FIFOs to reduce interrupt loading. Easier to just throw it from the disk as it comes, out of a reasonably wide port and let an FPGA sort out all the interleaving and critical timings .
« Last Edit: January 28, 2013, 12:02:36 am by mikeselectricstuff »
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26755
  • Country: nl
    • NCT Developments
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #44 on: January 28, 2013, 12:35:37 am »
You can still buy UARTs as a seperate chip. You just need 5 of these:
http://uk.farnell.com/exar/xr16v554ij-f/ic-uart-quad-16b-fifo-68plcc/dp/1716899

You can always opt for sending the data from a timer interrupt. If you know the baudrate then you know when its time to send the next bytes. A 33.3kHz interrupt is very doable on an ARM. I recently made a design which uses a 10kHz interrupt to send bytes over an SPI bus while the rest of the software may be busy on writing to an SD card. By the time the next timer interrupt comes the previous SPI transfer is done so its safe to dump the data into the TX buffer.

edit: I don't think you'll need an address/data bus. If you choose the databus as consequtive bits on a port you could 'bit-bang' it and still be fast enough. Maybe its even too fast for the UART to keep up...
« Last Edit: January 28, 2013, 12:48:58 am by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline Marco

  • Super Contributor
  • ***
  • Posts: 6694
  • Country: nl
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #45 on: January 28, 2013, 02:51:12 am »
Erich Styger has done quite a bit of open source work to make the Kinetis chips work smoothly with SDHC. Integrating FatFS (read only open source implementation of FAT) with Freescale's SDHC library into an eclipse component. Dunno what the throughput is though. One of many entries on the topic :

http://mcuoneclipse.com/2012/11/18/arduino-data-logger-shield-with-the-frdm-kl25z-board/
« Last Edit: January 28, 2013, 03:02:09 am by Marco »
 

Offline amyk

  • Super Contributor
  • ***
  • Posts: 8240
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #46 on: January 28, 2013, 09:32:08 am »
Is CF an option? The protocol is much simpler (basically ATA)  and the interface is 16 bits wide. You can have the microcontroller do directory traversal/file finding, and send the list of blocks in the file to the FPGA, to be stored in its RAM. It can then read the blocks itself and push the data out its UARTs, no additional filesystem overhead required so the latency is fixed; you can even clock the card synchronous to the data being output.

10Kbyte/frame * 50 frame/sec / 2bytes/clock = 250KHz CF clock.


The same idea could be done for SD although at higher clock frequency. Not possible with USB.
 

Offline mrflibble

  • Super Contributor
  • ***
  • Posts: 2051
  • Country: nl
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #47 on: January 28, 2013, 12:56:54 pm »
Maybe something like the CY7C67300 would do the trick?

Currently in stock at digikey/mouser.
« Last Edit: January 28, 2013, 12:59:17 pm by mrflibble »
 

Online mikeselectricstuffTopic starter

  • Super Contributor
  • ***
  • Posts: 13695
  • Country: gb
    • Mike's Electric Stuff
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #48 on: January 28, 2013, 02:34:22 pm »
Is CF an option? The protocol is much simpler (basically ATA)  and the interface is 16 bits wide. You can have the microcontroller do directory traversal/file finding, and send the list of blocks in the file to the FPGA, to be stored in its RAM. It can then read the blocks itself and push the data out its UARTs, no additional filesystem overhead required so the latency is fixed; you can even clock the card synchronous to the data being output.
Yes, quite possibly - not quite as small or ubiquitous as SD or USB, and more work on PCB layout but would be acceptable.
Small size is sometimes important  - I did one SD player using a QFN PIC24, which had the same outline as the MicroSD card. However higher channel counts are less likely to be space-constrained. 
The Microchip library does appear to support CF over the parallel master port peripheral so may be worth  a look. It's not supported on the Explorer devboard, so may not be as well tested or supported but may be worth a go.
May even be able to just passively skim off the parallel data stream into the FPGA.

Quote
Maybe something like the CY7C67300 would do the trick?
Not high speed.
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline firewalker

  • Super Contributor
  • ***
  • Posts: 2450
  • Country: gr
Re: Looking for high speed USB memory stick or SDHC reading solution
« Reply #49 on: January 28, 2013, 02:44:03 pm »
Just curious. You are using monochrome LEDs or RGB ones?

Alexander.
Become a realist, stay a dreamer.

 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf