Real time applications (think along the lines of streaming video) rely on the ability to supply new data faster than it is consumed, which means buffering a certain amount before beginning consumption, and being able to resupply the buffer quickly enough once you reach a "low water mark" to prevent exhaustion.
So depending on how quickly the FPGA is going to consume the data you send it, you'll have to determine what kind of bit rate you need to be able to supply data to prevent your buffer being exhausted, taking into account the headers that you need to send it as well.
I would not rely on real time streaming of the data to the FPGA over USB, as there is no guarantee about what kind of jitter you will encounter, so this will likely not result in a smooth supply of data. If you must go down this route, at the very least you should have a small buffer, but you'll need to do some experiments to determine how big that should be, or allow it to be configured.