FPGA high speed UDP stream and packet drops on PC

[radiolistener]

I need to receive and process high speed UDP stream from FPGA, the speed is about 400-1000 Mbps. But I catch some issue with packet drops on PC side.

When CPU load is zero, I can capture stream with test app almost with no packet drops up to 600-700 Mbps. But if CPU load grows up to 20-30%, I got a lot of packet drops. Also I got packet drops even at 85 Mbps if CPU load grows up to 40-50%.

I'm using QuadCore Intel Core i5-760 which has 4 cores running at 3200 MHz (overclocked) and Win7 x64 OS. Ethernet card is onboard 1G Ethernet, it is displayed as "Realtek PCIe GBE Family Controller" on the device list. The chip is probably Realtek 8112L, as said for my motherboard on some review site, but I'm not sure.

Packet drops happening at OS kernel level, so dropped packets never received from a socket. I tried to use multiple threads to call recvfrom(), but it leads to broken packets order, so I needs to analyze packets id and reconstruct original packets sequence before processing. It add even more CPU load and memory usage which is very critical, because I'm working with gigabit realtime stream with no pauses for processing.

I found that enter into realtime kernel mode (change OS kernel time slice to 1 ms with timeBeginPeriod) helps a lot to get rid of packet drops in a single threaded receiver app. But it also leads to 2-3 times higher CPU load, probably due to excessive overhead for thread context switch.

So, I wonder what is the best strategy to receive gigabit UDP stream with no packet drops on a usual PC?

[BradC]

If you're going to use it on commodity hardware you might need a different operating system that offers some better packet handling. Failing that you could try a better network card that has a decent on-board receive buffer and a driver that knows how to use it.

[radiolistener]

yes, it seems that there is definitely some issue with network card and OS performance, because when I push gigabit UDP stream from FPGA into PC and there is no app listening for that stream, the PC shows about 40-50% CPU load. This is CPU load with no running applications...

Is it possible to reduce that load with network card which has some hardware acceleration/buffers? What network card can you suggest?

Forgot to note, I'm using a small 1032 bytes UDP packets. I know it's better to switch for a large jumbo-packets in order to reduce streaming overhead and improve network performance. But it also requires additional OS/router configuration. So, I'm trying to use classic packet size.

And the question about receiving high speed stream is still active, what is the best algorithm to receive high speed UDP stream from a gigabit interface?

What is better - to use multi-threaded or single-threaded code to read the socket?

How much receive buffer memory should be assigned to the socket?

Any tricks to improve performance?

Any suggestions for driver precise tuning? (there are a lot of settings for ethernet driver in the device properties)

[BradC]

What network card can you suggest?

An enterprise grade Intel card would be my go-to. They tend to have on-board buffers, packet and interrupt coalescing and well supported DMA with drivers that are reasonably well tested. I'd avoid anything by Realtek. Even if you do get one that doesn't have massive errata to work around, the drivers are usually a steaming pile.

And the question about receiving high speed stream is still active, what is the best algorithm to receive high speed UDP stream from a gigabit interface?

Whatever works best for your scenario. Honestly it's very dependent on what you are doing with the data and how your OS manages packets. I know nothing about the Windows architecture because I don't use it for anything remotely arduous or critical. I did once, many years ago and learned a valuable lesson.

What is better - to use multi-threaded or single-threaded code to read the socket?

See above. Also depends on your software architecture.

How much receive buffer memory should be assigned to the socket?

Depends. If what you are doing is not latency sensitive, lots and lots. If it's latency sensitive then as much as you can reasonably tolerate.

Any tricks to improve performance?

Observation, instrumentation and profiling. Find out where the bottlenecks are by measuring them and hammering them out.

Any suggestions for driver precise tuning? (there are a lot of settings for ethernet driver in the device properties)

Nope. Entirely driver and device specific. I'd leave another disparaging comment about Windows, but you get the idea.

[dmills]

For really screaming network performance you are often better off forgoing interrupts entirely and just pinning a core to the task of polling the NIC for new data.

Needs an OS that lets you play those games of course, but the fact is an interrupt on an X64 is EXPENSIVE, in my case with a 10G interface running UDP I could not afford them so I just spun on the rx status register, 100% CPU load on that core, but so what?

Even at 1G you would benefit from a card that can do interrupt coalescing and maybe zero copy networking.

[bd139]

Going to keep this simple: don't use UDP if you don't want packet drops. There is no guarantee any packets will reach the destination and no way to know if they did or not or why they didn't. Even if it looks like it works today, a tiny kernel or environmental change somewhere may break every assumption you have.

Accept the packet loss in your design philosophy or use a different protocol.

[agehall]

While UDP isn’t reliable 100%, pushing 1Gb/s on a LAN using UDP shouldn’t be an issue if you have proper hardware. And by “proper” I mean pretty much any server grade NICs that have real offloading and interrupt coalescing.

[FenTiger]

Packet loss across the open Internet is always possible and is something you need to design for.

Packet loss across a single point-to-point Ethernet link is a symptom that something is wrong - either a hardware problem, or (as in this case) the receiver isn't keeping up.

it is displayed as "Realtek PCIe GBE Family Controller" on the device list

Some people would regard that as a hardware problem.

[bd139]

Oh Realtek. Yes I'd start there too 

[SiliconWizard]

While UDP isn’t reliable 100%, pushing 1Gb/s on a LAN using UDP shouldn’t be an issue if you have proper hardware. And by “proper” I mean pretty much any server grade NICs that have real offloading and interrupt coalescing.

You're right, but you still won't have any hard guarantee. If packet loss is unacceptable for the given context, then UDP is never going to work.
If you can accept some low rate of packet loss (TBD), then it's doable with proper hardware AND software.

As a workaround, if you manage to keep packet loss rate to a very low level, you could implement (if at all possible with your protocol) a simple packet resend scheme.

[BrianHG]

While UDP isn’t reliable 100%, pushing 1Gb/s on a LAN using UDP shouldn’t be an issue if you have proper hardware. And by “proper” I mean pretty much any server grade NICs that have real offloading and interrupt coalescing.

You're right, but you still won't have any hard guarantee. If packet loss is unacceptable for the given context, then UDP is never going to work.
If you can accept some low rate of packet loss (TBD), then it's doable with proper hardware AND software.

As a workaround, if you manage to keep packet loss rate to a very low level, you could implement (if at all possible with your protocol) a simple packet resend scheme.

Even in a project with a MCU in the past, using UDP required us to use multi-level checksum and packet retransmission on a regular basis.  Having different routers in our network environment could even make data loss and errors worse.  You better test whichever communication code you generate with an entire bunch of network hardware from great to garbage especially including WiFi as that networking hardware will put your com code through hell, specially at large distances or in a noisy environment.

[spaceballs]

Like others say, use NIC card with offboard logic, enterprise grade.

[radar_macgyver]

If you have the opportunity to dual-boot the Windows box in question, try booting a live-USB Linux distribution and run iperf in UDP mode. That will test your existing NIC. I haven't used a Realtek NIC, but getting zero packet loss for 6-700 Mbit/s over a 1G ethernet link is not difficult, unless there is a cabling issue.

[FenTiger]

Nobody has asked the OP what kind of network configuration he is using, or what other traffic is present at the same time.

I'll hazard a guess that it doesn't involve any WiFi, though.

[agehall]

While UDP isn’t reliable 100%, pushing 1Gb/s on a LAN using UDP shouldn’t be an issue if you have proper hardware. And by “proper” I mean pretty much any server grade NICs that have real offloading and interrupt coalescing.

You're right, but you still won't have any hard guarantee. If packet loss is unacceptable for the given context, then UDP is never going to work.
If you can accept some low rate of packet loss (TBD), then it's doable with proper hardware AND software.

As a workaround, if you manage to keep packet loss rate to a very low level, you could implement (if at all possible with your protocol) a simple packet resend scheme.

Of course - if you can’t accept any packet loss, UDP is never the right choice but on a LAN with modern hardware, 1Gb/s is not exactly a data rate where you would expect any issues at all, at least not in my network. But since OP choose UDP to begin with, I’m going to make the assumption that the inherent limitations of UDP are acceptable and in that case, some packet loss is not an issue. However, when using Realtek NICs, it doesn’t take much to make things fall over and I’m fairly certain that is the actual problem here.

[radiolistener]

Nobody has asked the OP what kind of network configuration he is using, or what other traffic is present at the same time.

I'll hazard a guess that it doesn't involve any WiFi, though.

Currently I'm using at 100-300 Mbps through home gigabit router, which includes Wi-Fi, but when I using it at higher speed, I'm using dedicated direct cable connection from FPGA board to PC. Because router is unable to handle such traffic. The router starts to choke at about 500-700 Mbps speed, it just stops to respond for all ports and starts dropping packets.

Regarding to resend the packet, there is no memory in FPGA to store packets. And since this is realtime stream, it will be impossible to resend the packet, because it will require extended network bandwidth, which limit is already reached. Also it requires complicated logic to store all packets and write it at about 125 Megabytes per second to the memory.

As I see, UDP transfer works pretty good on 100M ethernet link at max speed and there is almost no dropped packet at all. So I think it can work the same on 1G ethernet link. But there is an issue on PC side, for some unknown reason it just cannot handle so many packets. I have no idea why. Of course I was used direct cable connection with no intermediate devices for testing it.

Unfortunately I don't have other network adapters than realtek for testing it.

[BradC]

But there is an issue on PC side, for some unknown reason it just cannot handle so many packets. I have no idea why.

As politely as possible and paraphrasing the replies above : "Because you are using a crap NIC with a crap driver on a crap operating system".
If you really need the hardware to work at 100% then you need decent hardware with decent software backing it up.

[radiolistener]

As politely as possible and paraphrasing the replies above : "Because you are using a crap NIC with a crap driver on a crap operating system".
If you really need the hardware to work at 100% then you need decent hardware with decent software backing it up.

I'm not sure about that, because when I'm using windows file transfer between two PC it shows about 1G transfer speed. This is why I think that there is possible way to use it at 1G speed, but I just don't know how. Usual code examples for UDP listening have a lot of packet drops at 1G speed.

I set 256 Megabytes for socket receive buffer, but it doesn't help much. The only thing that I found which affects it is a time slice switch in OS thread scheduler with timeBeginPeriod call. When I call timeBeginPeriod(1) packet drops almost disappeared, but CPU load grows up very significantly...

[agehall]

Currently I'm using at 100-300 Mbps through home gigabit router, which includes Wi-Fi, but when I using it at higher speed, I'm using dedicated direct cable connection from FPGA board to PC. Because router is unable to handle such traffic. The router starts to choke at about 500-700 Mbps speed, it just stops to respond for all ports and starts dropping packets.

You seriously need to get at least half decent gear.

A router that chokes at 700Mb/s is either utter crap to begin with or seriously broken. (Might also be misconfigured and reject the UDP stream as some sort of attack) With a server class NIC and a proper router, or better yet, a good switch, this should not be an issue at all.

[SiliconWizard]

Many "low-end" routers have pretty limited CPU power and handle much of the routing in software, so you're never going to get the full throughput with them unless you have like zero routing rule and use them just as a switch.

The better ones have hardware routing capabilities which helps performance tremendously.

[hamster_nz]

What do your network statistics say? ("netstat -e" and "netstat -s") Any errors or checksum issues?

Are you using jumbo frames, rather than the "safe minimum" of 548 bytes of user data? This will reduce the OS overhead a lot.

Are you maintaining the required interpacket gaps? 12 idle bytes between packets?

What are you doing when you get a packet? Could it be distracting the OS long enough to cause the NIC ring buffer to overrun?

For a similar data acquisition issue I had to allocate a large memory buffer, write junk into each block so it was actually present, before receiving the data into the buffer. 

[FenTiger]

This is why I think that there is possible way to use it at 1G speed

This way is called TCP. It is very good at dealing with flaky networks. It is also very complicated. You don't want to try to implement it in an FPGA. Trust me on this.

[BrianHG]

This is why I think that there is possible way to use it at 1G speed

This way is called TCP. It is very good at dealing with flaky networks. It is also very complicated. You don't want to try to implement it in an FPGA. Trust me on this.

To get this performance, in TCP, they also use 'Packet Scheduling'.  Do not believe you can achieve this with an FPGA running simple UDP.  I'm not sure packet scheduling exists for UDP, or what layer of the network protocol where this is implemented.

[nightfire]

First, I would make sure that the hardware is up to the task.
Usually in this kind of load scenario, there are lots of interrupts to deal with, and this leads to latency and congestion.

A server grade Intel Card would also be my tip, as most of the other manufacturers are whether special and hard to get or difficult with drivers (Broadcom etc.)
Realtek cards have gotten better since the 10 Mbit Koax systems, but consumer Realteks are not designed for a real stressful environment.
Network layer: To really make sure everything is ok, I would go for a good quality connection, using CAT6 or better cables from a brand manufacturer, and use a dedicated "real" switch, not the stuff embedded in some small router. With a managed switch it ís also possible to watch for possible issues with the connection.
Also important: Depending on the type of switch/manufacturer, latency times can vary significantly for packet forwarding.

At work I use HP Procurve/Aruba 2530 for critical servers, something similar to that should be available used for decent money-so it is possible to rule out other reasons during transmission.

[radiolistener]

What do your network statistics say? ("netstat -e" and "netstat -s") Any errors or checksum issues?

No, there is no errors. But when packet drops starting, it shows that "Discards" value starts to grow very significantly. When there is no packet drop it shows "Discards" = 0. And "Errors" value still remains zero.

I'm not sure what "Discards" value means exactly. As I read here it shows "The number of packets rejected by the NIC, perhaps because they were damaged".

But that packet drops issue appears when CPU load is above some threshold. When there is no CPU load, there is no packet drops. So this is definitely not damaged or malformed packets.

By the way, most of my speed tests with UDP I performed with Alex Forencich verilog ethernet stack implementation, which seems pretty stable and tested for up to 10G and 25G speed.

Are you using jumbo frames, rather than the "safe minimum" of 548 bytes of user data? This will reduce the OS overhead a lot.

Hm... I didn't know about that "safe minimum" 548 bytes... Why 548 bytes?

I don't use jumbo frames, because it needs to enable it in network card properties. By default it is disabled on my driver. And as I remember when I enable it it affects network performance for a usual packets. So I'm using 1024+8 bytes packet, 1024 bytes for a data payload and 8 bytes for id and streaming control.

Are you maintaining the required interpacket gaps? 12 idle bytes between packets?

yes, 12 bytes gap. When I tried to reduce it, the network card just stops to accept the packets and as I remember I even don't see any packet in the Wireshark.

What are you doing when you get a packet? Could it be distracting the OS long enough to cause the NIC ring buffer to overrun?

Basically I send it to HDSDR program which performs FFT and further DSP processing. But for a cleaner test of network performance, I tried to receive data, check packet id and just drop it with no further processing. In both cases I have packet drops. So, it seems that data processing in HDSDR doesn't affect it much, the only note here is that data processing add some CPU load.

I tried two methods to receive the data:
1) using fixed buffer for socket recvfrom call, next I extract payload to another fixed buffer and send it to HDSDR. All this is done from the same thread with highest priority dedicated to the socket receive task.
2) using fixed buffer for socket recvfrom call, next I copy the packet content to another fixed buffer and enqueue it for processing in another thread which extracts data and sends it to HDSDR.

The first method has a smaller CPU load and less packet drops. Probably due to additional cost for a thread synchronization and queue processing.

Interesting thing is that change receive thread priority to normal sometimes can reduce packet drops. But when I added timeBeginPeriod(1) call it turns into opposite - highest priority helps to reduce packet drops.

Also I found that for some unknown reason, the packet drops are significantly reduced when I run receive process under debugger (MSVS), I have no idea why... When I thinking why it may happens I got the idea to try to change OS time slice with timeBeginPeriod. And it really helps to reduce packet drops very significantly. But as a side effect I got higher CPU load.

It seems that there is possible another some internal OS thread which is involved in a network card processing. So, playing with time slice interval and thread priority affects that hidden thread.