Digital FPV video for drone racing

[TheDane]

* The camera and its transmit-side electronics must be physically small and weigh well under 100 grams. A particular camera with which I'm familiar is 74 grams and considered grossly overweight by most pilots. Total aircraft dry weight is generally around 300 grams including video system, of which the four motors alone are 100-120 grams.

* Establish a low latency network (50mS is about the max) on an ISM band (so no licenses are required) that can support at least eight simultaneously active pilots.

http://socialcompare.com/en/comparison/raspberrypi-models-comparison
Raspberry Pi Zero W - without headers: 9 grams
Raspberry Pi Zero WH - with headers: 12 grams

Some random 5.8 GHz AC WiFi high speed dongle - https://www.amazon.co.uk/Daffodil-Ultra-Fast-Dongle-LAN05/dp/B016QKQ1U0#productDetails
Weight: 9 grams

5.8 GHz WiFi band is ISM, and has ping times at/below 1 mS if done correctly 

I do not know if the Raspberry Pi is good enough to handle video encoding at breakneck speeds - fortunately other SBC's exists that does not use Broadcom , such as the Beagle Bone(s)
(I have not used those to video-encode, so I can't comment on it's usability):
https://www.mouser.com/new/seeedstudio/BeagleBone-black-vs-green/
The TI SoC has a Programmable Real-time Unit (PRU) - making it excellent for controlling hardware pins, timing, transfers, etc.
It is large (it is a development board) - its weight is around 40 grams.

I would say hardware wise, it is not impossible - it should be rather light, inexpensive, and easy to obtain.
Software - well, can't help you there

Edit - clarify "5 GHz band is ISM, and has ping times at/below 1 mS if done correctly  "

[ogden]

Doing a partial frame update/interlacing - is like filling a FIFO buffer, and only rendering the final picture frame when full. Great for slow/non-moving stuff.

Most digital video sensors works same "scan line" way as oldskool TV - readout happens line by line. At 640x480 @ 60fps analog tranceivers will have roughly 1/60 sec = 16.666 msec "delay". Proposed 8-line buffer will add 8 * ( 1/(60*480) ) = 0.2777 msec delay. To me 1.66% frame time increase due to 8 line buffer does not look like much. Assuming that existing analog transceivers are ideal giving 16.66 msec best case latency, for digital video we use 120 FPS so summary interframe time + buffering delay is (16.666 +  0.2777)/2 = 8.472 msec. If we add kinda pessimistic 1 msec digital radio delay (doable even with WiFi), total latency is 9.472 msec. One is clear - radio is not the main contributor to (perceived) latency.

[IDEngineer]

One is clear - radio is not the main contributor to (perceived) latency.

Agreed. The discussions of latency here have centered on various proposed data processing steps such as compression to obtain higher resolution, or FEC to handle RF dropouts. Today's pure analog systems certainly don't have much in the way of processing latency, AFAIK they don't even buffer a line - it's as real time as the semiconductors make possible.

[TheDane]

Most digital video sensors works same "scan line" way as oldskool TV - readout happens line by line.

One is clear - radio is not the main contributor to (perceived) latency.

One thing is the sensor, another is the display.
The sensor output is 'serial' - the display output is 'not'.  I hope you get where I am pointing at, as the screen has to be 'stationary long enough to show the entire picture', and then show the 'entire next picture long enough'

I too agree on the radio tech being fast enough.

Edit: Adding info:
Oldschool TV's/CRT work 'in serial mode', however the screen used phosphor which has a persistence glow, thus 'keeping the image on screen'
https://en.wikipedia.org/wiki/Cathode-ray_tube#Phosphor_persistence

[ogden]

One thing is the sensor, another is the display.
The sensor output is 'serial' - the display output is 'not'.

Are you sure?

[TheDane]

It would be spamming doing a lot of replies:

S
o

Y
e
s
 
I

a
m

u
n
l
e
s
s

y
o
u
r

b       v
r        i     
a       s
i       i
n       o
        n

w
o
r
k
s

d
i
f
f
e
r
e
n
t
l
y

t
h
a
n

m
i
n
e

[IDEngineer]

I would say hardware wise, it is not impossible - it should be rather light, inexpensive, and easy to obtain.

Very promising, if the hardware can handle whatever processing is desired. Great way to prototype. Then a commercial solution could take the various components and integrate them onto a single small PCB, while eliminating all of the unnecessary headers and USB connectors. The only off-board connections are power (2) and camera (3, power plus video, so the camera cable need only go to a single point), presuming that the antenna has its own connector or soldered coax.

This is great, except for the following. Many configurations run the NTSC camera output through the (separate PCB) flight controller, and then the flight controller outputs NTSC to the video transmitter. This allows the flight controller to superimpose OSD text on the NTSC as I mentioned earlier in this thread. Pilot vision is entirely within the goggles, so they need OSD to monitor battery status and other real time flight data that the camera and video transmitter have no way of knowing. This is one reason why sticking to NTSC is quite important... the entire infrastructure of the sport, from on-aircraft avionics to the ground support monitors used by judges and crew, rely on the use of NTSC. I suppose this new proposed video transmitter could accept additional (analog?) inputs and do the OSD itself, but flight controllers would have to be redesigned to emit such signals AND that doesn't address the extensive existing ground equipment.

[ogden]

It would be spamming doing a lot of replies:

So you say that LCD does not take picture sequentially? - Meaning controller takes whole frame into RAM, then frame miraculously in no time appears on LCD screen? 

Just searched YT for Arduino LCD topic to show how LCD update happens in slo-mo:

https://youtu.be/X8syt55ITUo?t=267

[TheDane]

One thing is the sensor, another is the display.
The sensor output is 'serial' - the display output is 'not'.

Are you sure?

You're talking about the display INPUT, I presume
The output is being outputted, until new information is stored in display memory - and the screen refreshed - showing the new output for a 'long' time, much longer than  it took for the data to inputted

[ogden]

You're talking about the display INPUT, I presume
The output is being outputted, until new information is stored in display memory - and the screen refreshed - showing the new output for a 'long' time, much longer than  it took for the data to inputted

Before you spread your wizdom, make sure you got it right

[edit] I talk about this BS where you supposedly tell your beliefs about how LCD works:

I hope you get where I am pointing at, as the screen has to be 'stationary long enough to show the entire picture', and then show the 'entire next picture long enough'

[TheDane]

I would say hardware wise, it is not impossible - it should be rather light, inexpensive, and easy to obtain.

Very promising, if the hardware can handle whatever processing is desired. Great way to prototype.

This is great, except for the following. Many configurations run the NTSC camera output through the (separate PCB) flight controller, and then the flight controller outputs NTSC to the video transmitter.

Sorry to cut some of your quote - I hope it's ok.

I have been working with the MinimOSD myself, a mix of an Arduino and the MAX7456 OSD chip. It didn't start out that way 
http://ardupilot.org/copter/_images/MinimOSD_Micro.jpg

If a SBC is used to generate video (it interfaces directly to a digital camera), what is stopping it from measuring other vital flight information (Gyro, Battery voltage, etc) - which is then superimposed onto the generated video image - which is then sent to the pilots FPV goggles.
It is essentially a flight computer, capable of video+information handling - just like your existing system

[TheDane]

You're talking about the display INPUT, I presume
The output is being outputted, until new information is stored in display memory - and the screen refreshed - showing the new output for a 'long' time, much longer than  it took for the data to inputted

Before you spread your wizdom, make sure you got it right

[edit] I talk about this BS where you supposedly tell your beliefs about how LCD works:

I hope you get where I am pointing at, as the screen has to be 'stationary long enough to show the entire picture', and then show the 'entire next picture long enough'

How Not To Overclock:

[richard.cs]

Because you can't use inter-frame compression (due to latency)

If you use 8x8 blocks, the only necessary latency is buffering 8 lines ... 0.19 ms at 720p60.

8x8 blocks is intra-frame compression surely? Or have I got my terminology completely wrong here.

What I meant was that most of the redundant information in a raw video stream is in the time domain, the parts of the image that either do not change from one from to the next or change only by simple translations. Any compression algorithm that accounts for that has to delay by a full frame, and a compression algorithm that doesn't can't remove the information that is redundant in time.

Oh, I see what I missed. Because you're comparing your 8x8 block to the previous frame is does not introduce additional delay beyond waiting for 8 lines, or a little more if that block has translated vertically. Good point.

[ogden]

How Not To Overclock:

So what? That picture do not even apply to what we are talking about. Most LCD displays immediately show pixels as they arrive through interface. If you wonder what VSYNC and HSYNC signals are for - they are pointer/coordinate reset so both LCD and controller agrees where on the screen pixel contents will be going to. Watch YT video I mentioned

[Kilrah]

I do not know if the Raspberry Pi is good enough to handle video encoding at breakneck speeds

No it's not, there's a popular project using the combination of Pi+wifi and while it works it is not satisfactory in terms of latency. Main reason is that the whole video pipeline and GPU of the Pi is completely closed so there is nothing in there you can optimize, your only way through that is call the provided API function to get a frame from the "black box", with unnecessary buffering. Most SoCs on popular SBCs have the same problem of either closed libs and/or unobtainium documentation.

So you say that LCD does not take picture sequentially? - Meaning controller takes whole frame into RAM, then frame miraculously in no time appears on LCD screen? 

Just searched YT for Arduino LCD topic to show how LCD update happens in slo-mo:

https://youtu.be/X8syt55ITUo?t=267

That's a small, low performance SPI module, you just populate its internal buffer in the order you want and at the speed you want.
 
Normal fast high res displays (such as the ones with the common parallel 24bit RGB interface) of course take the data sequentially but also in order and at constant refresh rate, you can't just stop sending stuff while you wait for the next line or such, if you don't respect timings you get what is shown in the video above.
The standard way that computers and display controller boards use is to hold usually 2 full frame buffers that are sent to the screen alternatively using DMA at the screen clock aka out of sync with the incoming data so that the controller can populate the inactive buffer while the other is sent to avoid screen tearing and artifacts, but that obviously introduces 1-2 frames of latency.

Since the display panel is in the end the one that's driving the timing really minimizing latency would require lots of syncing so that the whole camera, encoder, transmitter and receiver work just right so the data arrives at the point where it is to be sent to the LCD as close as possible to the moment where it is to be sent to the display panel, with minimal buffering with only small FIFOs containing a fraction of a frame between each component instead of one or multiple frames. And some fast, logic-driven (hence FPGA again) that is capable of "filling in" the data to be sent to the display when a packet is missing at a small multiple of the pixel clock which is typically 25ns on a 800x480 panel.

That's what makes it hard to use off the shelf stuff. Almost all of it just puts buffers between each step so as not to have to bother with that, and it's why you get >100ms latency with most standard camera->computer->RF->computer->display chains.

[ogden]

Normal fast high res displays (such as the ones with the common parallel 24bit RGB interface) of course take the data sequentially but also in order and at constant refresh rate, you can't just stop sending stuff while you wait for the next line or such, if you don't respect timings you get what is shown in the video above.

Right. So this is essentially how analog TV updates screen. Only difference is digital input and "infinite persistence" due to built-in frame shadow buffer for refresh purposes (LCD's with built-in controller). I was opposing TheDane who implied that you shall put whole frame into LCD and only then it can be shown, which is untrue:

The sensor output is 'serial' - the display output is 'not'.  I hope you get where I am pointing at, as the screen has to be 'stationary long enough to show the entire picture', and then show the 'entire next picture long enough'

[Marco]

Hey all, I have decided that I'm going to put this project on hold on my end. You guys have brought up so nsny good points that i had not considered that make this project damn near infeasible. Feel free to continue discussion, as I feel it is very constructive.

Not infeasible, just a lot of work. Lets say you go with the approach I suggest, here's roughly how ignoring unforeseen roadblocks I'd chop up the tasks :

- get preempt_rt and camera working on SBC and find out how to get low latency access to the framebuffer (unfortunately the H3 boards I mentioned before won't support a 60fps camera, just use a Raspberry Pi 3 for now)
- get EZ-USB and TLC7528 DAC working with SBC
- create monochrome NTSC signal from camera input, which is not that hard. Feed it to the transmitter and you can fly at this point
- get EZ-USB and AD9200 ADC working with laptop running Linux with preempt_rt (easy to convert to SBC later on, but save yourself the headache for now)
- connect the ADC to the appropriate trace in the receiver, decode monochrome NTSC and put it on screen

Then you are ready to start experimenting with encoding, FEC, multi-path correction and compression. Then at the end of the line you can port the work to a lower weight SBC or even a FPGA.

[dmills]

Lots of confusion about how LCD display modules work.

The common graphics ones often have a few operating modes, typically you can have a microprocessor interface to a frame buffer, or an (often) 16/18 bit interface with Hsync and Vsync that may or may not involve the frame buffer depending on what you want to do, finally there is sometimes a microprocessor style option that also uses Vsync. MIPI, DSI and the like are basically Hsync/Vsync style interfaces just done over a serdes.

The tradeoff is that in non frame buffer mode you typically cannot do rotation in the screen itself, and quite a few of these screens are annoyingly setup for tall and narrow as their native scanning format. Obviously a rotation involves a frame of latency any way you cut it (and more typically two frames of latency).

For real low latency video there is still something to be said for a an old Sony BVM series grade 1 CRT....

73 Dan.

[ogden]

Marco just described complex, multimillion $ worth SDR development project like it's nothing. In short: if you have to develop digital radio transceiver - better drop whole "Digital FPV" project right now. If you can live with "off the shelf" digital radio ASIC (or module), then video processing part can be tried by bunch of dedicated developers with loads of free time.

unfortunately the H3 boards I mentioned before won't support a 60fps camera, just use a Raspberry Pi 3 for now

It is good idea to start with widely available stuff - rPI, it's camera and WiFi dongle to use it as prototype development system. BTW some forum chat says that rPI camera can run faster than 60FPS. Using existing software solutions/libraries you can build proof of concept remote display - to get taste of that huge pile you will be digging through.

Then at the end of the line you can port the work to a lower weight SBC or even a FPGA.

Right. I would plan FPGA from very beginning, use rPI just as development system for "low latency FPV video codec" and develop it with FPGA in mind. Video 480p or even 720p, >= 120FPS, camera with IMX219PQ or similar, FPGA (Zynq) for video compressor/decompressor, 5.8GHz 802.11ac (half-miniPCIe module) for transmission channel. When video development finished and it is worth to push radio latency into sub-1ms, *then* WiFi bells and whistles can be stripped down to just essential data transmission with minimal overhead, yet this alone is very complex task and most likely separate project on its own.

[Marco]

Marco just described complex, multimillion $ worth SDR development project like it's nothing.

You would be using an existing transmitter and receiver, you aren't touching anything over 6 MHz. The ADC/DAC's I mentioned are 1$ parts ... this is more equivalent to a RTLSDR software project than anything else.

Neither the existing transmitter nor receiver really cares about the signal being NTSC, that's why the original idea had merit.

Everything is so much more difficult in FPGAs, starting from attaching the camera. There's no real light weight Zynq boards and designing one is a big undertaking in and of itself. If you're doing compression on the ARM, what's the point of a FPGA any more any way? FEC does 100s of Mb/s on these kinds of processors, where exactly do you see the need for FPGA at all? The only thing a FPGA makes a little more compact is attaching the DAC/ADC, since you can forego the EZUSB workaround.

[ogden]

You would be using an existing transmitter and receiver, you aren't touching anything over 6 MHz. The ADC/DAC's I mentioned are 1$ parts ... this is more equivalent to a RTLSDR software project than anything else.

You most likely will *not* use existing transmitter and receiver, unless you want to have hilariously bad sensitivity specs or hilariously slow speed. Their SNR, linearity and phase noise specs are insufficient for high order modulations at high symbol rate resulting in > 2bits/1Hz spectral density. You keep persistently dreaming your uneducated dreams about plugging ADC and DAC into 6MHz baseband of video transceiver and easily demodulating 20Mbit or so, data with same ease as RTLSDR is showing spectrum and playing FM radio. All this to addition of video decompression using same CPU's BTW. You really shall study much more about what it takes to develop functioning hi-speed demodulator of digital radio.

Everything is so much more difficult in FPGAs, starting from attaching the camera.

Well.. obviously. After all you shall compress video coming at 640*480*120*2 = 73.7 MBytes second (120Hz, 16 bit color). 720p will require even higher bandwidth. H.264 video compressors of rPI uses GPU as coprocessor, but as I know, GPU of rPI is not Cuda compatible. So - good luck with rPI GPU development. I would love to see how easy as you say, is compression of such video stream using just ARM cores.

If you're doing compression on the ARM, what's the point of a FPGA any more any way?

Why would you suggest that in FPGA solution I offered video compression using ARM? It is kinda dumb to have FPGA but not use it

The only thing a FPGA makes a little more compact is attaching the DAC/ADC, since you can forego the EZUSB workaround.

That EZUSB workaround for ADC/DAC with SDR modem is no better than common USB WiFi dongle with proper radio/modem ASIC.

[Kilrah]

Well.. obviously. After all you shall compress video coming at 640*480*120*2 = 73.7 MBytes second (120Hz, 16 bit color). 720p will require even higher bandwidth. H.264 video compressors of rPI uses GPU as coprocessor, but as I know, GPU of rPI is not Cuda compatible. So - good luck with rPI GPU development. I would love to see how easy as you say, is compression of such video stream using just ARM cores.

The cores are unlikely to be fast enough especially not given the latency requirement (can't make good use of the multiple cores by spliting tasks), and as said the Pi's video pipeline / encoder and GPU are "black boxes" so you can't do fancy stuff with them like encoding a 8px high line from the frame at a time, you've fot to feed it one of the standard supported frame sizes as a whole.

But while a high res full frame encoder takes a lot of FPGA fabric we have the advantage that since we precisely want to encode small portions the encoder block can likely fit in a smaller area.

[Marco]

20Mbit or so

That's about an order of magnitude off from the last bandwidth I mentioned for video compression.

All this to addition of video decompression using same CPU's BTW.

Compression tends to be the harder part.

H.264 video compressors of rPI uses GPU as coprocessor, but as I know, GPU of rPI is not Cuda compatible. So - good luck with rPI GPU development. I would love to see how easy as you say, is compression of such video stream using just ARM cores.

It might be possible to encode slices with the videocore without adding too much latency (part of the OpenMAX API AFAICS). So say encode 128 lines at a time as a slice, that way you don't have to queue up an entire frame. Would require some experimentation. Still easier than getting video coding to work on a FPGA.

That EZUSB workaround for ADC/DAC with SDR modem is no better than common USB WiFi dongle with proper radio/modem ASIC.

It fulfils the design parameters set in the OP, in that it is better.

[TheDane]

Who needs sync signals every line when you have crystals? You can throw a reference pulse in there every couple of ms and calculate the path response and correct for it, etc.

Crystals are NOT perfect, and will change frequency when subjected to g-changes (acceleration, deceleration, rotation). Ah relativity 

Late last night I remembered about MEMS - Microelectromechanical system oscillator - https://en.wikipedia.org/wiki/Microelectromechanical_system_oscillator
When compared to a crystal, a MEMS oscillator is relatively immune to random vibration and has much better phase and jitter specs.

This might be a good replacement for the existing crystal oscillators on already flying drones, to improve video quality?

Some tests and conclusion - page 10 is about random vibration:
www.sitime.com/api/gated/AN10032-Shock-Vibration-Comparison-MEMS-and-Quartz-Oscillators.pdf


Oh, just don't use them with your helium filled blimps - they will crash

[ogden]

20Mbit or so

That's about an order of magnitude off from the last bandwidth I mentioned for video compression.

2Mbit was not just example but actual offer of data bitrate? 640*480*120*2=73MByte/sec is 584 Mbits/sec coming out of camera. Even half that (8bit grayscale) is huge number. You offer to compress video with fast vertical panning using 292:1 compression? - You sound like those 21-st century teens who declare that there's no need for education/experience/skills other than trained thumbs and internet

I have to remind that 120fps h.264 cannot reach better latency than analog TV transceivers (60Hz half frames). By 100% saturating data channel you are adding one more frame time, so absolutely best delay of such h.264 120FPS system would be 3/120=1.5/60 sec while analog NTSC TV currently have better latency of 1/60 sec!