The death of Xon/Xoff flow control...

[magic]

I took a quick glance at the CDC spec and its' all about stuff like "telephones", "modems", "ISDN", etc.

I would guess that if flow control was not standardized then maybe those things simply didn't need it, and if they didn't need it then USB had no reason to bother standardizing it.

[Karel]

Funny exercise on windows:

Connect a microcontroller/device to a windows pc using a usb/rs232 converter. Feel free to use the latest and
greatest converter using highspeed usb (480Mb), bigger buffers, etc. Also, use the fastest pc or workstation
available with whatever version of windows since XP.

Let the device send a contnuous stream of data at 115K2 baud to the pc that uses some protocol to check for lost
bytes/characters.
Use some kind of test application on the pc that reads the incoming data from the virtual comport and just checks
for lost bytes/characters.

Disable any flowcontrol.

Don't use the pc for any other tasks and make sure it's not updating in the background.

You'll probably notice that no characters will be lost.

Now, using the mouse, grab the downright corner of a window of some application and start to quickly and
continuously resizing that window. Notice what happens with your incoming data....
Spoiler: lots of bytes lost caused by buffer overruns.

Conclusion: windows gives approx. 100 times more priority to graphical updates then to interrupts from the
usb hostcontroller chip on the motherboard of the pc. Also, you cannot have reliable rs232 communication
on windows without using hardware flowcontrol ( rts/cts) when the baudrate is more than 9600 baud.

The same is applicable to Linux but much, much less severe than windows.

[PlainName]

Conclusion: windows gives approx. 100 times more priority to graphical updates then to interrupts from the
usb hostcontroller chip on the motherboard of the pc.

Yes, that's by design because if it were the other way around users would complain about laggy display and similar. Windows for servers has a setting to invert that (because they are typically headless) so that background tasks take priority over the UI.

[IanB]

Funny exercise on windows:

Connect a microcontroller/device to a windows pc using a usb/rs232 converter. Feel free to use the latest and
greatest converter using highspeed usb (480Mb), bigger buffers, etc. Also, use the fastest pc or workstation
available with whatever version of windows since XP.

Let the device send a contnuous stream of data at 115K2 baud to the pc that uses some protocol to check for lost
bytes/characters.
Use some kind of test application on the pc that reads the incoming data from the virtual comport and just checks
for lost bytes/characters.

Disable any flowcontrol.

Don't use the pc for any other tasks and make sure it's not updating in the background.

You'll probably notice that no characters will be lost.

Now, using the mouse, grab the downright corner of a window of some application and start to quickly and
continuously resizing that window. Notice what happens with your incoming data....
Spoiler: lots of bytes lost caused by buffer overruns.

Conclusion: windows gives approx. 100 times more priority to graphical updates then to interrupts from the
usb hostcontroller chip on the motherboard of the pc. Also, you cannot have reliable rs232 communication
on windows without using hardware flowcontrol ( rts/cts) when the baudrate is more than 9600 baud.

The same is applicable to Linux but much, much less severe than windows.

But Windows is not a real-time operating system, so giving priority to the GUI is to be expected.

However, what happens if the same experiment is tried, but first increasing the process priority of the test program from the task manager, even to real-time priority?

[coppice]

But Windows is not a real-time operating system, so giving priority to the GUI is to be expected.

Sure, Windows is not a real time system, but its priority scheme is a mess. You can get things so badly scheduled that they clog your machine so it takes minutes to kill the offending task. The GUI can make your media stutter. Media is a key real time aspect of a modern GUI that Windows handles very poorly.

[Nominal Animal]

The same is applicable to Linux but much, much less severe than windows.

I haven't experienced that on Linux at all.  My preferred test method is generating a Xorshift64* random number sequence with seed sent first by the Linux application, using only 32 high bits of each generated 64-bit number (because that is one of the rare ones that pass all BigCrush tests; not even Mersenne Twister does).  When testing MCU-to-host, the host-side application generates the same sequence and compares it to the one it receives from the MCU.

As an example, Teensy 4.0 can sustain over 200 Mbits/s or 25 Mbytes/s to Linux hosts using USB CDC ACM, "USB serial", when writing in blocks of 32 bytes or larger.  Depending on the machine and load it varies between 21 and 29 Mbytes/s, with 25 Mbytes/s typical and expected, lower end only seen on some slow SBCs under heavy load.  I've tested this for hours on end, and seen no lost data.

If you lose USB data packets, it is a bug, either on the device or the host, either hardware or in the USB stack implementation.  USB is fundamentally not supposed to lose packets at all, because of the ACK/NACK responses.  AFAIK, all models of Raspberry Pis are known to have this fault, and it is only mitigatable in software, not fully fixable at all.  I do not know of any other Linux SBC that has this fault.

At 119200 baud, you use less than 12 kbytes/s.  To starve an application of this bandwidth, your process priorities must be set up wrong (i.e. deliberately niced too much), even on slow SBCs.  Even an ATmega32u4 (as in Pro Micro clones) can sustain over 900 kbytes/s, almost 1 Mbyte/s, over USB CDC ACM, continuously.

Thus, no.  I definitely have not seen this on Linux at all, outside of Raspberry Pis which have buggy USB hardware, not even on Linux SBCs or minilaotops.  It is purely a Windows and Raspberry Pi issue in my opinion and experience.

[free_electron]

You may have to disable the hardware fifo in the usb<>serial. the xoff is hidden in the fifo buffer, so you don't see it until the fifo is purged. By that time a new packet is in the fifo... arrived while you were purging. A CDC transport(assuming that is what you are using) is still 512 bytes with each frame being 64 bytes.

[Nominal Animal]

You may have to disable the hardware fifo in the usb<>serial [for XON/XOFF flow control on the host to work].

Yes, but that will absolutely kill the bandwidth.  At FS (12 Mbit/s), you'll max out the USB bus at about 1000 bytes/second – one byte sent per send slot opportunity; at HS (480 Mbit/s), at about 8000 bytes/second.  Neither suffices for 115200 baud.

[magic]

Connect a microcontroller/device to a windows pc using a usb/rs232 converter.

As an example, Teensy 4.0 can sustain over 200 Mbits/s or 25 Mbytes/s to Linux hosts using USB CDC ACM.

Thus, no.  I definitely have not seen this on Linux at all, outside of Raspberry Pis which have buggy USB hardware, not even on Linux SBCs or minilaotops.

That's a slightly different thing, isn't it?

[Karel]

[
If you lose USB data packets, it is a bug, either on the device or the host, either hardware or in the USB stack implementation. 

Who is saying you loose usb datapackets???

The mechanism is this:

- the device sends a continuous stream of data to the host (a windows pc)
- windows gets busy with something "more important"
- the rx buffer of the usb hostcontroller gets full because the os is not reading fast enough from it
- the usb host controller does not accept new datapackets from the usb slave (actually, it will not honour requests from the slave to upload data)
- the tx buffer of the usb slave gets full (rs232/usb converter chip) because it cannot and will not send datapackets when the hosts rx buffer is full
- the device keeps on writing new bytes into the rs232/usb converter chip because cts/rts are either not connected or ignored
- the rs232/usb converter chip either ignores the new bytes for tx or overwites the old bytes in the tx (fifo)buffer, hence the lost bytes

[iMo]

Have you confirmed that the system sending data does not respond to XOFF at all?  Ie - set up a second terminal emulator and typed your XOFF manually?  You could consider sending XOFFs pre-emptively, like at intervals half of what you'd expect to fill up whatever buffering you do have...

Yes, I have tried everything. The XOFF characters are simply ignored by the PC side.

Hmm, as I wrote above, when I issued ctrl-Q/ctrl_S from the keyboard (Win10, two TeraTerms against each other via FT232 dongles) the other side (transmitting side) stopped/started even at 921k properly..
Also the files I transferred were ok (sent as "binary" half duplex), while I saw the xon/xoff chars coming from the receiving side.
PS: while watching the xon/xoff channel I saw on o'scope the "C8 and 88 hex" chars coming in irregular intervals (those are the xon/xoff chars in reverse bit order when looking at it from the start bit). Perhaps somebody with a better o'scope with large mem may track those xon/xoff chars during a longer transfer and see how the buffering actually works..

https://www.eevblog.com/forum/general-computing/the-death-of-xonxoff-flow-control/msg5584567/#msg5584567

[magic]

No, Karel demonstrated that disabling flow control on Windows pretty much guarantees data loss, which means that Windows fails to schedule the consumer process (or the USB driver) often enough to drain hardware RX queues before they overrun.

XON/XOFF flow control does work on USB adapters. I have just tested with two PL2303 connected to each other. I was able to transfer a 10MB text file through the connection without any error, ten times. Without flow control, it would often fail after a few megs.

To enable (on Linux):

Code: [Select]

stty -F /dev/ttyUSB0 ixon
This functionality appears to be implemented in the USB-UART chips themselves, so it should work with low latency. Not all drivers implement it, for example PL2303 and FTDI do, but CH431 does not.

[m k]

This whole implementation is pretty silly, it's like updating display with a CPU.

Maybe the real problem was that Microsoft is a software company.
They did a legacy free PC, so obviously all *external* legacy *hardware* was also off.
The goal was that USB is the bridge, and it was, but other end had legacy software.
So obviously rest of the world had to react, I can still remember first CLI/STI tests, only few hundreds per second.

So legacy hardware was connected to legacy software through USB and obviously old stuff was tried to keep in the play.

Back then Microsoft had power and authorities didn't really understand.
When W95 came Microsoft cheated and changed things just before release.
So all external application software makers were lagging.
Before that Lotus 1-2-3 and WordPerfect were major players.

Hardware side was a bit different, but not much.
All industrial machines had to maintain their connections.
Parallel side was easy, just stream those same bytes over the USB.
Serial could have been the same, who messed it?

My guess is that the reason is Microsoft again.
They excluded device drivers from free compilers.
So big part of driver updaters were kicked out.

Back then parallel was already leaving behind, but serial was all around.
Changing some old external parallel devices was pretty easy when changing serial ones was a practical impossibility.
Parallel was beside you and serial somewhere else, and possibly very exotic, but also very vital.
An example, punched tape, YeeYee port connected punching machine was already left behind and direct serial added.

Next obstacle was SMB 1.0 but that's a different story.

[nctnico]

Have you confirmed that the system sending data does not respond to XOFF at all?  Ie - set up a second terminal emulator and typed your XOFF manually?  You could consider sending XOFFs pre-emptively, like at intervals half of what you'd expect to fill up whatever buffering you do have...

Yes, I have tried everything. The XOFF characters are simply ignored by the PC side.

Hmm, as I wrote above, when I issued ctrl-Q/ctrl_S from the keyboard (Win10, two TeraTerms against each other via FT232 dongles) the other side (transmitting side) stopped/started even at 921k properly..
Also the files I transferred were ok (sent as "binary" half duplex), while I saw the xon/xoff chars coming from the receiving side.
PS: while watching the xon/xoff channel I saw on o'scope the "C8 and 88 hex" chars coming in irregular intervals (those are the xon/xoff chars in reverse bit order when looking at it from the start bit). Perhaps somebody with a better o'scope with large mem may track those xon/xoff chars during a longer transfer and see how the buffering actually works..

The problem is that the USB adapters need to implement flow control in their drivers which is a hit &miss. And as Nominal Animal has researched, in USB-CDC UART flow control is broken even worse. The basically leads to a situation where UART flow control is highly hardware & software dependant which in turns means you can't rely on flow control to work using whatever hardware you have, nowadays. In your case you are just lucky it worked for you.

[magic]

It's more hit than miss unless your hardware is garbage, but software setup is tricky.

You say you tried with minicom. I found that this thing disables flow control (of all kind) when it starts and I'm not sure if that can be overriden in any way. But you can turn it back on after starting minicom, and then it works.

But yeah, probably not something you would want random customers to do.

edit
The problem of CDC limitations is resolved very easily: no one seems to make USB-RS232 cables based on CDC. I have only ever seen it used in devices which get data straight from USB packets and only pretend to be behind a serial port. CDC appears to be designed specifically for such devices and flow control built into USB bulk endpoints is all they need. Actual RS232 bridges use proprietary protocols.

[free_electron]

The problem is that the USB adapters need to implement flow control in their drivers which is a hit &miss.

it's a hardware issue. if the hardware could detect xon/xoff this problem would not exist.
- You are trying to run a protocol on top of a protocol on top of a protocol. ( xon on top of uart on top of usb ).
- add to that you are stuffing packets in packates (packet delineated by xon xoff on usb packets in frames.
- the usb hardware adds latency and double buffering. if a single character is received uart side it needs to be stuffed ina  64 byte usb frame.

this results in the xon/xoff characters to be somewhere in the stream.. by the time you receive them it is too late : there is more stuff already on the way. the same goes for outgoing xon xoff.. it is completely asynchronous with an unknown latency.
The only option is to make sure you have a buffer large enough to hold at least two complete usb frames (512 bytes). if the first frame contains the xon xoff you can react and there possible is another 512 on the way. taking into account the outgoing latency i would make it 3 frames. you may be sending while a second packet is in flight so it will take another 512 bytes at the opposite side to guarantee them trapping it and stopping transfer.

that is your only option. large buffers. or downprogram the buffer size in the uart. the ftdi devices have an early flag you can set on one of the pins that tell you the buffer has reached a threshold. you can tie that to an interrupt on the micro. ping the micro that the buffer is 1/3 or 1/4 full and the micro can send the xoff. anything in flight will still land in the built-in buffer

[nctnico]

The problem is that the USB adapters need to implement flow control in their drivers which is a hit &miss.

it's a hardware issue. if the hardware could detect xon/xoff this problem would not exist.
- You are trying to run a protocol on top of a protocol on top of a protocol. ( xon on top of uart on top of usb ).
- add to that you are stuffing packets in packates (packet delineated by xon xoff on usb packets in frames.
- the usb hardware adds latency and double buffering. if a single character is received uart side it needs to be stuffed ina  64 byte usb frame.

this results in the xon/xoff characters to be somewhere in the stream.. by the time you receive them it is too late : there is more stuff already on the way. the same goes for outgoing xon xoff.. it is completely asynchronous with an unknown latency.
The only option is to make sure you have a buffer large enough to hold at least two complete usb frames (512 bytes). if the first frame contains the xon xoff you can react and there possible is another 512 on the way. taking into account the outgoing latency i would make it 3 frames. you may be sending while a second packet is in flight so it will take another 512 bytes at the opposite side to guarantee them trapping it and stopping transfer.

that is your only option. large buffers. or downprogram the buffer size in the uart. the ftdi devices have an early flag you can set on one of the pins that tell you the buffer has reached a threshold. you can tie that to an interrupt on the micro. ping the micro that the buffer is 1/3 or 1/4 full and the micro can send the xoff. anything in flight will still land in the built-in buffer

No. As I wrote before, I tried all this as I have full control over the device which is receiving the data. It is the PC side which isn't dealing with xon/xoff properly and that can't be fixed in a way it will always work.

[Andy Chee]

Ben Eater released a new video, adding hardware flow control to his breadboard 6502.  No mention of USB issues, though perhaps his 6502 isn't running particularly fast baud rates.

[Nominal Animal]

The mechanism is this:

Ah, now I understand.  No, the rx buffer of the USB host controller does not get full (increasing it won't help), it is just that the host will not ACK the packet before the USB interrupt has occurred, so the device runs out of internal buffer.

Your complaint is that in Windows and in Linux that interrupt has too much latency for typical RS-232/UART to USB converters.

At 115200 baud, the data rate is about 11520 bytes per second.  If the converter to-host buffer size is N bytes beyond one USB packet –– remember, the converter cannot discard the sent USB packet until it has been ACKed, or indeed data may/will be lost ––, the allowed interrupt latency is N/11520 seconds, or N·86.8µs.  Let's say a typical converter has 128 bytes of buffer (so 64 bytes over one full packet at FS), which corresponds to 5.6ms.  Essentially, it is double-buffering the device-to-host USB packets.

Typical desktop systems often have longer interrupt latencies than that 5.6ms under load, because the scheduling unit is of that order.

In Linux, you can mitigate this by using a kernel configured to use 1000 Hz ticks (CONFIG_HZ=1000, CONFIG_HZ_1000=y).  Debian desktop (and derivatives like Ubuntu and Mint) defaults to 250 Hz, so the scheduling "unit" is 4ms.

The question is, how much buffer space is reasonable on any USB device?  That is, what kind of USB packet ACK latencies one should expect in practice?

In my opinion, triple buffering (or 192 bytes at USB 2.0 FS, 1536 bytes at USB 2.0 HS) should be the absolute minimum for any kind of pipe/bus with receive acknowledge.  I typically count on at least 4 USB packets: 256 bytes for FS, 2048 for HS.  More if available, of course.

This means that in my opinion, a "typical" 128-byte buffer is unrealistically small.  Especially when they claim to support 1 Mbaud (or higher!) rates, which corresponds (with a 128-byte buffer) to 0.56ms maximum USB packet-acknowledge interrupt latency.

Feel free to choose which one you blame: the OS for having USB packet ACK latencies longer than a couple of milliseconds, or the converter devices for having ridiculously small buffers for what they claim to do.  There are very valid reasons for both.

[Nominal Animal]

Do note that USB to serial converter chips that use their own driver instead of USB CDC ACM can properly support RTS/CTS and XON/XOFF flow control (on the converter chip itself).  It is only the USB CDC ACM standard/class driver that cannot.

[IanB]

Ben Eater released a new video, adding hardware flow control to his breadboard 6502.  No mention of USB issues, though perhaps his 6502 isn't running particularly fast baud rates.

I watched that video.

He was trying to paste large amounts of text into the 6502-based computer from a terminal program on the PC.

He used the hardware CTS/RTR pin on the microcomputer side to tell the PC to stop sending when the micro's UART buffer was nearly full, and resume when the UART buffer had more space. He set the terminal program on the PC to use RTS/CTS flow control.

It worked fine to avoid lost data. I think the success of this might have depended on what USB/RS232 adapter the PC was using. I have not watched the earlier videos to find that out.

[magic]

It works.

Shown below, blue is TX sending a stream of zero bytes and yellow is RX waiting for XOFF. As soon as that is received, the UART finishes the current byte and stops transmitting. The whole process takes nanoseconds, which certainly means that a round trip over USB to the CPU and back is not involved. This is PL2303, probably not even a genuine one.

FTDI would likely still send one more byte, because that's what it does when CTS/RTS flow control is used. This makes FTDI unusable with primitive MCUs with one byte RX buffer.

[magic]

The mechanism is this:

Ah, now I understand.  No, the rx buffer of the USB host controller does not get full (increasing it won't help), it is just that the host will not ACK the packet before the USB interrupt has occurred, so the device runs out of internal buffer.

Your complaint is that in Windows and in Linux that interrupt has too much latency for typical RS-232/UART to USB converters.

Of course a USB HC will ACK packets without CPU involvement, that's its damn job. How software can screw up is by not queuing enough transfer buffers - when the HC runs out of those, it will stop polling the endpoint for new data until more buffers are submitted by software.

[westfw]

no one seems to make USB-RS232 cables based on CDC.

It has become a thing.  People don't like installing "drivers" from "foreign" websites, that may or may not end up breaking existing setups (FTDI-gate?)  They would rather the dongle "just work with the standard drivers included with windows."

WCH, and I think SILabs or Cypress, have introduced new USB/Serial chips that have CDC/ACM support instead of custom drivers.  (eg https://www.wch-ic.com/products/CH9102.html )

[nctnico]

It works.

Shown below, blue is TX sending a stream of zero bytes and yellow is RX waiting for XOFF. As soon as that is received, the UART finishes the current byte and stops transmitting. The whole process takes nanoseconds, which certainly means that a round trip over USB to the CPU and back is not involved. This is PL2303, probably not even a genuine one.

FTDI would likely still send one more byte, because that's what it does when CTS/RTS flow control is used. This makes FTDI unusable with primitive MCUs with one byte RX buffer.

I don't think so. On such MCUs you'll have an interrupt + software ring buffer to receive data. If you use flow control, you set the high-watermark at a certain fill level of the ring buffer (say somewhere between 50% and 90%) which means you can still accept some extra data.