The death of Xon/Xoff flow control...

[nctnico]

I'm in a bit of a pickle with a project which needs to receive large amounts of text (more than anticipated) but it can't keep up with processing and buffering is not practical. So as a quick fix I thought to implement xon/xoff handshaking. Much to my surprise this doesn't work. First I wondering I got it wrong or sending the XON/XOFF characters at the wrong time. I tried several terminal programs like minicom, Cutecom and Putty but none seem to work. So I started to wonder if something is broken at a fundamental level. And this seems to be the case. Neither Linux and Windows have working xon/xoff handshaking implemented. More precisely, the xon/xoff function is handed to the driver and (next to) none of the hardware manufacturers seems to have implemented this form of flow control. This sucks because now I have to resort to using Xmodem.

This might not be news to many but I happened to stumble upon it today.

[nctnico]

I need a universal solution which is guaranteed to work on any platform and with any type of serial port. FTDI also has its gremlins BTW.

[iMo]

While looking at the FTDI VCP drivers release notes - the xon/xoff bug was fixed in 2013
So, perhaps those chips work..

[m k]

Seems that my memory doesn't include an UART with data line handshake.

Even DEC had it by the software.
VT100 had 8080 and the standard wasn't really a standard yet.
VT220 had 8051, so hardware was finally a good second.

PC had 8250 and its power was less than more, and no memory.
Maybe it was a competition thing.
Maybe it just wasn't needed.

Shouldn't be a very complicated project though.

[pqass]

What are you connecting with what?  Something like...
[project board]-----RS232(TX/RX/GND)-----[RS232-to-USB]----USB----[host computer]

If your complaint is that the host (running minicom ascii file send) is sending too much data and not responding to XOFF from your project,
then (in minicom) check Ctrl-A Z O serial port G, and/or
in another xterm check "stty -F /dev/ttyUSB0 -a".
       [-]ixoff   enable sending of start/stop characters
       [-]ixon   enable XON/XOFF flow control

Of course, your project must send an XOFF before its buffer is full and XON when its buffer goes empty.

For additional info see "Flow control and blocking I/O" here. It specifically speaks to the scenario of slow ttys connected to fast hosts.

[magic]

I have no experience using this. I have only used HW flow control (CTS/RTS) and it works OK as long as the UART and the driver support it.

I wonder about one thing: XON/XOFF is often called "software flow control", so isn't it implemented by SW? If so, how is it affected by RX buffering in USB UART chips? Isn't it the case that XOFF gets stuck in the buffer and by the time the SW receives it, it has already TX'd too much?

[IanB]

Every time I have used RS-232 communications, XON/XOFF flow control was something that had to be enabled with a flag or switch at both ends. It is not turned on by default because it is only really intended or useful for terminal equipment with keyboard and screen, and only at low bit rates like 9600 or 19200. XON/XOFF could be valid characters in binary data and would therefore mess up the communications in other applications.

In the majority of cases hardware flow control (RTS/CTS) is superior and should be used instead.

You might also consider using the DSR/DTR signals, which sometimes are implemented, and can be used to indicate whether a device is online/offline and whether or not it is ready to receive data.

[nctnico]

What are you connecting with what?  Something like...
[project board]-----RS232(TX/RX/GND)-----[RS232-to-USB]----USB----[host computer]

If your complaint is that the host (running minicom ascii file send) is sending too much data and not responding to XOFF from your project,
then (in minicom) check Ctrl-A Z O serial port G, and/or
in another xterm check "stty -F /dev/ttyUSB0 -a".
       [-]ixoff   enable sending of start/stop characters
       [-]ixon   enable XON/XOFF flow control

Of course, your project must send an XOFF before its buffer is full and XON when its buffer goes empty.

For additional info see "Flow control and blocking I/O" here. It specifically speaks to the scenario of slow ttys connected to fast hosts.

I wish it was this simple. After some Googling and reading it turns out that a lot of USB-serial port drivers don't support xon/xoff flow control and terminal software relies on the OS layers to implement xon/xoff flow control properly. The board is sending xon/xoff character just fine and when software flow control is enabled in any one of the terminal emulators, these characters are being absorbed by the OS / driver layer but no actual flow control is happening. So it all ends there as there is no fixing this problem in a way it will work for every situation.

[iMo]

..I wonder about one thing: XON/XOFF is often called "software flow control", so isn't it implemented by SW? If so, how is it affected by RX buffering in USB UART chips? Isn't it the case that XOFF gets stuck in the buffer and by the time the SW receives it, it has already TX'd too much?

I've wired two FTDI USB/UART dongles (232R and 232H) against each other and tried with the xon/xoff text transfers. Frankly, I cannot say whether it works or not. Using two Teraterms 5.1, one is the sender and the other one is the receiver. I do not see the xoff/xon chars from the receiver (watching the led) coming to the sender with any scenario, but it is not easy to simulate one slower and one faster device as the both dongles are on the same PC..
I will look at that tomorrow with oscope, whether it somehow does the handshake or not (the led blink is too fast to see with a single char at 921k8N1)..
Btw., there are infos in various places the FTDI USB UART xon/xoff handshake works with slower devices (ie with plotters).

[pqass]

I wish it was this simple. After some Googling and reading it turns out that a lot of USB-serial port drivers don't support xon/xoff flow control and terminal software relies on the OS layers to implement xon/xoff flow control properly. The board is sending xon/xoff character just fine and when software flow control is enabled in one of the terminal emulators, these characters are being absorbed by the OS / driver layer but no actual flow control is happening. So it all ends there as there is no fixing this problem in a way it will work for every situation.

Maybe switching to a RS232-to-USB adapter with a working driver will be the most expedient solution.

From here:   "I can confirm this problem using the ch341 driver. However using FTDI-driver detecting a FTDI232RL chip flow control with XON/XOFF works ..."

I connected a real VT220 to my PL2303-based RS232-to-USB adapter and confirmed CTRL-S from the terminal halted a large ASCII file send from minicom.  Although, I'm not sure why a CTRL-Q didn't resume it.

[thm_w]

..I wonder about one thing: XON/XOFF is often called "software flow control", so isn't it implemented by SW? If so, how is it affected by RX buffering in USB UART chips? Isn't it the case that XOFF gets stuck in the buffer and by the time the SW receives it, it has already TX'd too much?

I've wired two FTDI USB/UART dongles (232R and 232H) against each other and tried with the xon/xoff text transfers. Frankly, I cannot say whether it works or not. Using two Teraterms 5.1, one is the sender and the other one is the receiver. I do not see the xoff/xon chars from the receiver (watching the led) coming to the sender with any scenario, but it is not easy to simulate one slower and one faster device as the both dongles are on the same PC..
I will look at that tomorrow with oscope, whether it somehow does the handshake or not (the led blink is too fast to see with a single char at 921k8N1)..
Btw., there are infos in various places the FTDI USB UART xon/xoff handshake works with slower devices (ie with plotters).

You'd have to manually set xon/xoff and see if the data stops, then release it to see if the data resumes. You can't just send data at the same rate the PC can receive it, I don't think.

In putty you might be able to press ctrl+s: https://superuser.com/questions/124845/can-you-disable-the-ctrl-s-xoff-keystroke-in-putty

[iMo]

 
So I did again (2:30am here) - FT232R against FT232H, both 9k6 8N1 xon/xoff handshake.
TeraTerm 5.2 latest, a Win10 latest on different PC.
TT_A sends 11MB loremipsum to TT_B with "Send File".
Trying to Ctrl-S and Ctrl-Q in TT_B..
It works, it stops (S) and it resumes (Q) the receving in TT_B.
Did it 3x and got Blue screen with "DPC WATCHDOG VIOLATION".

Restarted and installed TT 4.105.
The same as above, the second Blue screen in last 15 years perhaps

Restarted and looked at the FTDI drivers - both drivers from Jan 8th 2013 (installed by the Win10 automatically, btw).
EDIT: my Win10 image was from 2018 upgraded to latest couple days back, so the old buggy driver comes from that 2018 image, imho..

Installed latest FTDI drivers manually.
Restarted and tried again (TT 4.105).
The Ctrl-S and Ctrl_Q work.

Set 921k8N1, and it still works..

While looking at the FTDI VCP drivers release notes - the xon/xoff bug was fixed in 2013
So, perhaps those chips work..

[m k]

Some DEC controllers had serial ports connecting only Tx, Rx and GND.
Then ReGIS was also a thing.

One other thing is communication cable length.
Hardware handshake is a constant signal level.

Back in the day handshaking was also "very close" to actual transmission line.
For graphics functions there were different MCUs.

Speed has nothing to do with handshaking per se, but 1/92k seconds for a character is not a very long time.
And being a single character to the other direction is counter intuitive, even that the speed is actually pretty slow.

[BrianHG]

Careful trying to use an FTDI com port with synchronous handshaking.  If the FTDI is transmitting, what you send to it during this time must be synced/framed with what you are receiving, otherwise, it may drop the first character you send in return.

It's been a decade since I played with this, but, a good test is to wire 2 PC, you master who has x-on/x-off enabled null modem wired to a second PC with a dumb terminal in 8N1 mode.  See if you can get you master to pause sending the ascii text on the second dumb terminal with while sending the control codes.

If the 2 PCs work, then your UART in the peripheral you are designing isn't generating a properly 'famed' TXD output and your X-on/X-off handshaking transmission is being ignored.


If you are using a homemade cheap public domain FPGA UART HDL, you will need to use something like my FPGA Synchronous UART to prevent this sync bug when attempting realtime handshaking.

[nctnico]

Xon/xoff are normal ASCII characters (DC1 and DC3) and send as part of the regular data. If a UART isn't handling those, then the UART won't work at all.

[PlainName]

Don't know about Xon/off availability but are you in control of the protocol? If so, take a leaf from TCP and implement a sliding window of appropriate buffer size. Or use Zmodem if you want a ready-rolled protocol.

[Bicurico]

Xon/xoff is/was heavily used by CNC machines when using the DNC protocol, which basically sends a big NC program to the CNC controller, for direct execution, because the controllers memory is smaller than the program to execute.

Most USB Serial port adapters do not work for this task, which is why companies still using DNC rely on old computers with a proper serial port.

Also, xon/xoff can be implemented by software using the two mentioned characters or by means of, if I remember correctly, pin 4 and 5 on DB25. This not support by most USB adapters, either.

The reason you need the hardware and/or driver handle xon/xoff is because this is toggled in regards of the buffer.

If you are stuck with a USB serial interface, your best bet is to reduce the baud rate so that you don't need xon/off.

[nctnico]

Don't know about Xon/off availability but are you in control of the protocol? If so, take a leaf from TCP and implement a sliding window of appropriate buffer size. Or use Zmodem if you want a ready-rolled protocol.

I have used Xmodem instead at the cost of quite a bit of extra code. I need things to work with any reasonable standard terminal emulator for simplicity.

[IanB]

I have used Xmodem instead at the cost of quite a bit of extra code. I need things to work with any reasonable standard terminal emulator for simplicity.

While reading through this thread, I never quite got to understand the application.

If I understood correctly there is some piece of bespoke hardware ("board") with a serial port, and it uses this to communicate with a user via a terminal or terminal emulator? But then you say that large amounts of text are involved. Does that imply upload/download operations to the device, or is it simply that the device sends vast amounts of data to the display (terminal or terminal emulator), which is faster than the terminal can keep up with? And you want flow control so that the screen doesn't glitch?

Which hardware do you have control over? Is the board a black box from a third party, or is it a board you have designed/implemented? And is the flow control required by the display terminal, by the board, or by both?

I know you said you have reached a resolution, but it was hard to follow where the problem exists. Like, does the terminal end send XOFF to the board to stop sending, but the board does not respond to the XOFF character? Or the board responds to XOFF/ON when it receives them, but the terminal does not send them?

And the question of RTS/CTS hardware flow control did not get much discussion.

At the end of the day, I don't think the terminal emulator is meant to do flow control. I think the terminal emulator program is supposed to configure the serial port with the right settings, and the serial port driver is meant to do the heavy lifting.

[IanB]

I did a quick search of Windows documentation on serial ports. I found this (link below):

A COM port controlled by Serial.sys is assigned a device name. A user-mode application can open this port by name, and then send I/O requests directly to the port.

...

Another difference is that Serial.sys implements software flow control, but Sercx2.sys does not. Both Serial.sys and Sercx2.sys support hardware flow control using the request to send (RTS) and clear to send (CTS) signals. For more information about flow control, see SERIAL_HANDFLOW.

Note that Serial.sys is intended for use with COM ports by terminal emulators and similar applications. Sercx2.sys is intended for high speed, point-to-point connections on circuit boards in hardware.

A terminal emulator using Serial.sys should be able to use the Windows API to configure the COM port with the required settings, including which flow control to use.

If it is a virtual COM port provided by a USB adapter, then it will depend on the features of that device. But the common hardware signals were provided on a USB serial adapter I have used.

[nctnico]

I have used Xmodem instead at the cost of quite a bit of extra code. I need things to work with any reasonable standard terminal emulator for simplicity.

While reading through this thread, I never quite got to understand the application.

If I understood correctly there is some piece of bespoke hardware ("board") with a serial port, and it uses this to communicate with a user via a terminal or terminal emulator? But then you say that large amounts of text are involved. Does that imply upload/download operations to the device, or is it simply that the device sends vast amounts of data to the display (terminal or terminal emulator), which is faster than the terminal can keep up with? And you want flow control so that the screen doesn't glitch?

The problem is not the board but the PC with the terminal emulator. The PC sends a large amount of text to the board. But the board can't keep up due to writing the text to flash. So as there is no way to implement hardware flow control, my idea was to use software flow control (xon/xoff). Only to find out the implementation of software flow control at the PC side is not guaranteed at all (which is the problem) so I needed a different solution (like Xmodem).

[pqass]

There is the Linux pv command (-L switch) that will rate limit a byte stream through a pipe.  Whenever you need to send a file, call a script that cats your file through pv and appends to the tty.

However, I think using XMODEM is probably the better way to go because there is inevitably a missing/bad byte every few K of data sent over RS232.  Simple and dumb XMODEM protocol has the advantage of small packets (132 bytes) with no sliding window so should give acceptable throughput over a low-latency link (host to board separated by meters).  A one byte checksum is still better than none but you might want to look at backward compatible XMODEM-CRC for extra protection.

[BrianHG]

One thing about FTDI USB converters, I did a project a few years back and flooding the transmit buffer with more than 2048 bytes could drop bytes even with hardware handshaking.  Even though the FTDI drivers supposed to have 4096 byte FIFOs.

One pacing method I used was a simple delayed echo, though depending on how you handle this, you can cut your bandwidth in half.  But logic wize, this used the least amount of cells in my FPGA handling the IO and still offered data verification.

[IanB]

The problem is not the board but the PC with the terminal emulator. The PC sends a large amount of text to the board. But the board can't keep up due to writing the text to flash. So as there is no way to implement hardware flow control, my idea was to use software flow control (xon/xoff). Only to find out the implementation of software flow control at the PC side is not guaranteed at all (which is the problem) so I needed a different solution (like Xmodem).

OK, that is clearer. But terminal emulators only send data as fast as someone can type, so that should not be a problem. Therefore, perhaps we are considering a file upload situation?

It seems if you want a reliable solution, you should have control over the PC side of things. Perhaps by writing bespoke software or defining the software to be used, and by requiring the use of validated hardware?

Also, if the board can't keep up with the incoming data, and the board does not have hardware flow control enabled, that would seem like a problem with the board.

[Andy Chee]

But terminal emulators only send data as fast as someone can type, so that should not be a problem. Therefore, perhaps we are considering a file upload situation?

Most terminal emulators are now able to "cut & paste" text directly into the window.  So it cannot be strictly called a file upload situation, but it is actually the equivalent of someone typing extremely fast into the terminal buffer.

(Reminds me of the Stargate SG-1 episode Upgrades, where Samantha Carter speed types several scientific papers, having a break now and then to allow the computer keyboard buffer to empty)

[IanB]

Most terminal emulators are now able to "cut & paste" text directly into the window.  So it cannot be strictly called a file upload situation, but it is actually the equivalent of someone typing extremely fast into the terminal buffer.

Good point. I honestly didn't think of that scenario 

[AndyC_772]

The problem is not the board but the PC with the terminal emulator. The PC sends a large amount of text to the board. But the board can't keep up due to writing the text to flash.

I've come up against a similar issue when using a serial interface to download firmware updates to a target.

To address the issue, my preferred way to handle serial Rx on the target is to use DMA. All received characters go into a circular buffer in RAM, which happens entirely in hardware.

This means no characters are dropped, even if the CPU stalls while a Flash write is in progress. As soon as the write completes and the CPU resumes operating, the code can process whatever characters were written into the buffer in the meantime.

[voltsandjolts]

nctnico is writing to flash, presumably not enough RAM available to buffer, hence the bottleneck.

Maybe use 300 baud connection! 

[AndyC_772]

Writing to Flash isn't that slow; I'd be amazed if the serial interface is faster than the mean write throughput. The problem is usually that if the CPU is running code from Flash, it stalls completely while writes are taking place, which means it can't run a UART Rx interrupt or poll the small FIFO in the UART.

DMA, however, can still work even when the core is stalled.

[iMo]

After some experiments above (TeraTerm+2xFT232) I wonder how to actually use the xon/xoff for the text transfer (==plain text).

When I send a file as a text the 0Ah 0Dh are not transferred. It works when sending the file as "binary". But that would not work with xon/xoff in duplex mode, imho.

Btw., in above setup in simplex transfer I see C8h and 88h bytes (on o'scope) coming from the receiving part to sender, no idea what they are for..
  THose are the xon/xoff bytes in reverse bit order, it seems..

[Andy Chee]

Something out of left field:

You haven't told us your broader application.  Perhaps it's your coding approach that could be improved upon, in order to execute fast enough within the time it takes to receive a serial byte at 115200 baud, for 8,N,1 format would be about 87uS.

For example, are you attempting to parse a text file for a particular string pattern or character?

Have you tried slowing the baud rate down to see if the problem goes away?  If the problem remains with slower baud rates, then it highly suggests a coding issue.

[mikeselectricstuff]

Writing to Flash isn't that slow; I'd be amazed if the serial interface is faster than the mean write throughput.

If it's SPI flash, erase times are a killer.

[Nominal Animal]

Apologies for the long post, but it should be worth the read for anyone dealing with fully-featured OS and microcontroller communications using USB Serial (CDC-ACM).

USB Serial (aka USB CDC-ACM) supports a few useful PSTN requests that are supported by generic OS drivers:

• GET_LINE_CODING (0x20) and SET_LINE_CODING (0x21) transfer a zero value with a 7-byte data record with
  • 32-bit baud rate (bits per second), in little-endian byte order
  • 8-bit stop bit count (0=1 stop bit, 1=1.5 stop bits, 2=2 stop bits)
  • 8-bit parity type (0=none, 1=odd, 2=even, 3=mark, 4=space)
  • 8-bit data bit count (5, 6, 7, 8, or 16)
  , allowing e.g. Linux drivers/usb/class/cdc-acm.c driver to maintain the corresponding termios fields.  See include/uapi/linux/usb/cdc.h:struct usb_cdc_line_coding for a C definition of this structure.
• SET_CONTROL_LINE_STATE (0x22) sends a 16-bit value from host to device, with all but the two lowest bits zero.
  • Bit 1: This corresponds to RS-232 RTS (Ready-To-Send) state.
  • Bit 0: This corresponds to RS-232 DTR (Data Terminal Ready) state.
  The generic Linux CDC-ACM driver sends this message automatically with bit 0 set when the tty device is first opened, and cleared when last open descriptor closes, allowing the MCU to check if there is an application on the host computer accessing the tty device or not.
  (In Teensyduino, the USB (Serial) object evaluates to a boolean corresponding to the bit 0, and is thus true if the tty device is open on a Linux host.)
• SEND_BREAK (0x23) with value corresponding to termios tcsendbreak() with the duration specified in milliseconds.

Similarly, there is one useful notification:

• SERIAL_STATE (0x20), with zero value but a 16-bit little-endian data, with the lowest seven bits having useful information:
  • Bit 6, bOverRun: Set if device discarded data due to buffer overrun
  • Bit 5, bParity: Set if device detected a parity error
  • Bit 4, bFraming: Set if device detected a framing error
  • Bit 3, bRingSignal: State of ring signal detection on device
  • Bit 2, bBreak: State of break detection on device
  • Bit 1, bTxCarrier: Corresponds to RS-232 DSR (Data Set Ready)
  • Bit 0, bRxCarrier: Corresponds to RS-232 DCD (Data Carrier Detect)

The generic Linux CDC-ACM driver maps SERIAL_STATE to TIOCMSET/TIOCMBIC/TIOCMBIS ioctl()s on the open tty device, and maintains the termios structure for the tty device via SET_LINE_CODING; see drivers/usb/class/cdc-acm.c:acm_tty_ioctl() and acm_tty_set_termios().

As you can see, there is no CTS (Clear To Send) support at all!

This is because the USB device is supposed to NAK the bulk data packets when it is not ready to receive more data yet.  Thus, a suitable USB Serial converter should respect the handshaking protocol using the abovementioned USB properties.  When internal buffers are close to full, host USB data packets are NAK'ed, and the downstream device is told to stop sending more data (via XOFF if software flow contro, via CTS=0 if hardware flow control).  That is, all flow control should be implemented in the USB-UART bridge, because USB CDC-ACM just cannot do it fully (not without vendor extensions).

As it happens, Stefan Wagner (wagiminator, I like his work) has a WCH CH551/2/4-based OLED project (sources, hardware whose terminal-mode usb_cdc.c (sdcc, so C99/C11/C23 and not C++ code) looks straightforward to modify for a software-or-hardware flow control USB-UART converter.  This way, the software side treats the USB connection as a normal bulk bidirectional data stream without any flow control; the CDC-ACM driver manages the USB flow control; and the USB-UART converter handles the serial flow control.  The part count is ridiculously low and cheap.

The Silicon Labs AN758 application node contains most of the useful USB info needed, but note it specifically does not implement or describe implementing any flow control.

For USB CDC-ACM protocols, one should note that the native FS packet size is 8, 16, 32, or 64 bytes depending on the endpoint configuration, and 512 bytes for HS.  USB 2.0 is inherently half-duplex, and while you can easily reach about 950,000 bytes/second in one direction, requiring a response packet for each one will decimate the bandwidth.  XMODEM is unfortunate in that it uses 132-byte packets, which requires an extra FS packet.  A custom protocol would yield somewhat better bandwidth, but because of the data ack/nak response packets, would not reach anywhere near what USB-UART converter handling the flow control can (because no response data packets are needed for flow control).

[DiTBho]

ttys* still has auto-clean on open, so if you open /dev/ttys0 the driver issues a CTS pulse, which may reset the target

Useless and annoying...

[nctnico]

XMODEM is unfortunate in that it uses 132-byte packets, which requires an extra FS packet.  A custom protocol would yield somewhat better bandwidth, but because of the data ack/nak response packets, would not reach anywhere near what USB-UART converter handling the flow control can (because no response data packets are needed for flow control).

Thanks for the investigation. Fortunately, there is also Xmodem 1k which uses 1024 byte packets which is widely supported and needs (ballpark) 3 lines of extra code to implement.  All in all the throughput is better using 1024 byte packets.

A dedicated tool is also not ideal because somebody has to pay for developing it and realistically, a terminal emulator already has this functionality. In addition I want to be able to give this in the hands of people not so tech savvy and they may run whatever OS which then ends up in another software product which needs maintenance, support and so on.

[m k]

Nowadays input has at least three wheels rolling on their own.
First UART, its job is to change serial to parallel.
Then memory, to remember those parallel characters.
And then USB, that does the move on part.

UART and memory must be hard wired, but USB can't be.
So the memory must be so big that it can hold an USB glitch.
Or handshake must be controlled by the hard wired part.

Back in the day DTR and DSR pins were for information of operational state of a device, so steady style.
Actual handshaking was meant to go through RTS and CTS.

DTE is a Data Terminal Equipment, practically everything we use now.
DCE is a Data Carrier Equipment, like old style modem and very obsolete.

DTE -- DCE  DTE -- DTE
DTR > DTR  DTR > DSR
DSR < DSR  DSR < DTR
RTS > RTS   RTS > CTS
CTS < CTS   CTS < RTS

But PC changed that and practically all handshaking combinations are possible.

Terminal emulations.
VT100 is 7 bit machine.
VT102 is 8 bit machine.

(if memory serves)
101 is 102 without a printer port.
125 is 100 based graphical terminal.
132 is 102 based editing terminal.
131 is 132 without a printer port.

Remember also that DEC machines are ANSI terminals by nature.
(DEC was the source, ANSI made standards)
So tapping a key can produce quite many characters.

Old Esc[ is later replaced with CSI, ESC with 8th bit set.
Characters 0xC8 and 0x88 have no special data flow meanings in early DEC machines, so perhaps some TeraTerm internals.

Binary full duplex xon/xoff mode is a misunderstanding, xon/xoff is not binary communication anything.
When binary file is sent the duplex is limited, second file can't go to the other direction simultaneously.
It's the same with all binary communications, there smallest communication part is more than smallest part of communication.
In UART's case the smallest is a bit, it can't be blocked inside a character.
Single signal level is not a binary communication.

A side note.
DEC LA34 and LA120 console terminals, matrix printers with keyboard.
Printing process is iron pin to color ribbon to tractor paper on iron plate.
Early model is slow, but it doesn't really soften its sound.
Today hearing protection would most likely be mandatory pretty fast.

[iMo]

FYI- a decade back a group of people around the retrobsd project spent a lot of effort with an attempt to run ZMODEM on the pic32MX based 2.11bsd unix (for an easy transfer of sources and bins to the unix). There are several various source codes available for zmodem, we tried perhaps with all (we build for the unix), with all possible parameters (there are many). Also there are many zmodem apps for the PC side. The best working one I found at that time is below (provided somebody is experimenting with zmodem). But generally, it never was 100% reliable. We used USB_UART dongles at that time (with various pic32MX boards).

Code: [Select]

Executing C:\Users\User\Desktop\Utility\winkermit-3.0-test-21-nov-2013-vc6\k95.ini for WIN32...
?SET SPEED has no effect without prior SET LINE
Good Morning, smile!
Kermit 3.0.0 Dev.-, Nov 21 2013, for 32-bit Windows
 Copyright (C) 1985, 2013,
  Trustees of Columbia University in the City of New York.
Type ? or HELP for help.
[C:\Users\User\Desktop\] K-95>

[m k]

I tried to understand those cdc-acm.c and cdc.c files.

Line 290 of cdc-acm has acm_process_notification.
From line 326 it's
if (difference & USB_CDC_SERIAL_STATE_DSR) acm->iocount.dsr++;
if (difference & USB_CDC_SERIAL_STATE_DCD) acm->iocount.dcd++;

DCD is Data Carrier Detect, it's a modem signal indicating that phone line is live.
So this section is for DTE.

From line 346 start status bits for UART, so everything is checked, but CTS is missing.
The thing is also adding counters, clearly uncertain that it can catch all changes.
But it's a wake up, possibly not so high priority.

Line 505 has acm_read_bulk_callback
From line 326 it's
case -EPIPE: set_bit(EVENT_RX_STALL, &acm->flags); stalled = true;

That is much like a flow control situation of receive buffer filling.

Line 579 has acm_write_bulk
It doesn't have anything about stalling transmit.
But it's from different level, so transmit buffer check must be elsewhere.

Line 665 has acm_port_dtr_rts
It has a comment that overall it's not very operational.
Indicating that DTR or RTS is not supported way too regularly.
Combining this with RX_STALL one can think that input side flow control is not very steady thing.

Line 929 has acm_tty_tiocmget
There TIOCM_CTS is external compared to other status signals.
Next is acm_tty_tiocmset
There TIOCM_CTS is nowhere.
So DTR and RTS out and DSR in seems to be supported and CTS not.

Line 998 has wait_serial_change
There DSR, RI and CD are checked.
All are sort of higher level modem signals, so CTS can be elsewhere.

Line 1096 has a speed check.
If speed is rational DTR is set.
Since RTS is missing one can think that DTR has a higher level, and lower level RTS is for flow control, if available.

cdc.c defines CTRL_DTR and CTRL_RTS first and then SERIAL_STATE_DSR among others.

8250 UART has
Modem Control Register bits 0 and 1 for DTR and RTS, and
Modem Status Register bits 5 and 4 for DSR and CTS.
It was also a standard for decades.

No idea when or why CTS has changed its availability.

So it's pretty clear that RTS to CTS cabling may have some problems.
Alternative is RTS to DSR, if DTR to DSR is not accepted.
Finally there are no difference between different status pins.
From UART's point of view they are just its internal register bit sources.
Its operational bits are elsewhere.

One other thing, a practical certainty, at least used to be.
One must be lucky if 8 data + any parity will work.

E,
Maybe flow control part wasn't clear.

One pin is sending and one receiving.
If you can control both ends it doesn't matter what is what.

Normally DTR and DSR are a pair, like are RTS and CTS.
But nothing prevents mixing them.
So one end can have DTR/DSR flow control when other has RTS/CTS.
It's just transmitter pin to receiver pin.

One disconnecting problem there was, but can't remember exactly what.
Early PC and its application software stuff anyway.
Maybe RTS/CTS flow control and DTR looped back to DSR and CD.
Then DTR blinks for what ever reason and used program disconnects because CD is lost.

[westfw]

There's Kermit (sliding window, designed for "difficult intermediate data paths.")  It should be supported by many "terminal emulator" apps.
I've suspected that "software flow control" has been on the "wrong side" of assorted buffering since "deep fifo" UARTs became available (ie, if you send a ^S to an OS that does XOFF in the driver to a UART with a 16byte FIFO, you have to expect at least an additional 16bytes of data.)  Some UARTs support XOFF in the UART itself, but I don't think it's that common.  It's probably a lot worse with USB/Serial converters involved - buffering on the USB connection plus buffering in the HW UART...
Have you confirmed that the system sending data does not respond to XOFF at all?  Ie - set up a second terminal emulator and typed your XOFF manually?  You could consider sending XOFFs pre-emptively, like at intervals half of what you'd expect to fill up whatever buffering you do have...

[Nominal Animal]

No idea when or why CTS has changed its availability.

The issue with USB CDC is that it already has its own "flow control", in the underlying USB transport itself.

In practice, this means that USB hardware and software flow control needs to be implemented in the USB device, and not in the USB CDC host drivers.

More annoyingly, the USB CDC specification does not have a standardized way to pass the necessary control messages between the USB device and the USB host, unlike say RTS and DTR.
This means that standard USB CDC drivers cannot properly support RTS/CTS (hardware) or XON/XOFF (software) flow control!

I do now believe it is a serious flaw myself.

The obvious solution would be to simply extend the SET_CONTROL_LINE_STATE and GET_CONTROL_LINE_STATE USB CDC PSTN control messages, as they have ten unused bits (specified to be zero), with the two lowest bits controlling RTS and DTR already.  In Linux, it would be rather easy to create a derivative driver from the generic one, adding this functionality – that is, when syncing termios settings between host and device, the handshaking controls would be included.

(For software flow control in all cases, users do need to remember to use the termios tcflow(descriptor,TCIOFF/TCION), and not just use XON/XOFF characters within the data stream.  And the ability to change the XON (VSTART) and XOFF (VSTOP) codes would need additional 16/32 bits depending on the maximum data bits supported, so likely a completely new control message.  Which isn't a big deal, though.)

With such a solution, the USB CDC device state would reflect the host termios state including software and hardware flow control and the XON/VSTART and XOFF/VSTOP codes.  The device-host flow control would use standard USB "flow control" that USB devices use to throttle the data flow.

I've been trying to understand how the original USB CDC specification intended to solve this problem, but cannot –– unless they thought that the handshake mechanism would be built into the USB device and be immutable.  Which is an utterly silly idea, considering it had until that point always been a mutable control, something that has been adjusted to suit each situation.  Thus, I consider this a fault in the USB CDC PSTN specification.

[nctnico]

Have you confirmed that the system sending data does not respond to XOFF at all?  Ie - set up a second terminal emulator and typed your XOFF manually?  You could consider sending XOFFs pre-emptively, like at intervals half of what you'd expect to fill up whatever buffering you do have...

Yes, I have tried everything. The XOFF characters are simply ignored by the PC side.

[m k]

What your mode command reports?

Though no idea what some of that actually means, but testing is easy.
(copy [something big] com1:)

Status for device COM1:
-----------------------
    Baud:            1200
    Parity:          None
    Data Bits:       7
    Stop Bits:       1
    Timeout:         OFF
    XON/XOFF:        OFF
    CTS handshaking: OFF
    DSR handshaking: OFF
    DSR sensitivity: OFF
    DTR circuit:     ON
    RTS circuit:     ON

I've been trying to understand how the original USB CDC specification intended to solve this problem, but cannot –– unless they thought that the handshake mechanism would be built into the USB device and be immutable.  Which is an utterly silly idea, considering it had until that point always been a mutable control, something that has been adjusted to suit each situation.  Thus, I consider this a fault in the USB CDC PSTN specification.

Maybe the thought was that since USB is a packet thing its outer side control is for other devices.
And then other bean counters decided differently, it's so fast and all.

[Nominal Animal]

I've been trying to understand how the original USB CDC specification intended to solve this problem, but cannot –– unless they thought that the handshake mechanism would be built into the USB device and be immutable.  Which is an utterly silly idea, considering it had until that point always been a mutable control, something that has been adjusted to suit each situation.  Thus, I consider this a fault in the USB CDC PSTN specification.

Maybe the thought was that since USB is a packet thing its outer side control is for other devices.
And then other bean counters decided differently, it's so fast and all.

It seems that the root cause is that RTS and CTS in RS-232 were designed to control a half-duplex connection, and "reusing" it for flow control happened later, and was never considered by USB CDC.

CDC PSTN 1.2 provides control messages for host to set DTR (Data Terminal Ready) and RTS (Request To Send) states.  The device can (but does not need to, this is optional) send SerialState asynchronous notifications, consisting of 16 bits with 15..7=0; bit 6 indicates received data has been discarded due to overrun; bit 5 indicates a parity error has occurred, bit 4 indicates a framing error has occurred, bit 3 indicates RI (Ring Indicator) status, bit 2 indicates Break detection, bit 1 indicates presence of Tx carrier (Data Set Ready, DSR), and bit 0 the presence of Rx carrier (Data Carrier Detect, DCD).

I do not know if the Windows generic CDC ACM reacts to SerialState notifications.

The Linux CDC ACM does; see drivers/usb/class/cdc-acm.c:case USB_CDC_NOTIFY_SERIAL_STATE.  If DCD is lost, the tty is hung up (ie. HUP signal if canonical mode, just like when a session closes).  The others are noted and obtainable via struct serial_icounter_struct using TIOCGICOUNT ioctl().  I have not tested whether manipulating DSR makes the host throttle/pause sending data.  Also, the TTY layer is supposed to throttle sending more data when termios IXON is set and c_cc[VSTART] and c_cc[VSTOP] are the IXON/IXOFF characters, but I haven't tested this either.  I probably should

Of the outgoing signals, DTR is supposed to reflect whether an application has the serial port open or not, and this works at least in Linux perfectly: the host tells the USB device it is set when a process opens the char device without any other process having it open, and it is cleared when the last open descriptor to the char device is closed.  The device USB CDC implementation just needs to parse SetControlLineState messages (and respond to GetControlLineState messages) from the host.

[PlainName]

It seems that the root cause is that RTS and CTS in RS-232 were designed to control a half-duplex connection, and "reusing" it for flow control happened late

Why should using it for full duplex be a problem? One end wants to send and uses RTS, it can receive and indicates that with CTS. Other end does the same and it works fine regardless of duplex mode.

But one thing that's missing is DTR/DSR. These are just as important for the protocol but since you can get away with ignoring them everyone does now. So perhaps part of the problem is that only half of the protocol is normally used.

Re USB: do you see the RTS/CTS equivalents on Ethernet? So why would you need them for similar packet-based protocol on USB? Fake com ports on USB would surely be at the same level as fake com ports via TCP/IP in a sensible world.

[nctnico]

I've been trying to understand how the original USB CDC specification intended to solve this problem, but cannot –– unless they thought that the handshake mechanism would be built into the USB device and be immutable.  Which is an utterly silly idea, considering it had until that point always been a mutable control, something that has been adjusted to suit each situation.  Thus, I consider this a fault in the USB CDC PSTN specification.

Maybe the thought was that since USB is a packet thing its outer side control is for other devices.
And then other bean counters decided differently, it's so fast and all.

It seems that the root cause is that RTS and CTS in RS-232 were designed to control a half-duplex connection,

No, most definitely not. RS232 has always been a full-duplex connection. What you have to keep in mind that back in the mainframe days, a mainframe had a whole bunch of terminals attached to it. Up to hundreds. All this data needed to be processed. Even though the RS232 interface boards inside the mainframe had local processors to offload the main CPU from dealing with low level stuff, a lot of data needed to be shuffled around. At the terminal side, you'd have a similar problem. Many terminals used rather nifty setups where the screen refresh was driven by some kind of interrupt / dma scheme using the same memory as the processor (typically some kind of 8051 on the common terminals). So the terminal side also needed a means to tell the mainframe to stop / start sending data. Requiring RTS/CTS/DTE/DTR  (or XON/XOFF) flow control to work was very real for those systems.

[coppice]

It seems that the root cause is that RTS and CTS in RS-232 were designed to control a half-duplex connection, and "reusing" it for flow control happened later, and was never considered by USB CDC.

RS-232C was designed for attaching modems to things. Data Set Ready, Data Terminal Ready, and Ring are a bit of a give away there. However, I agree that by the time USB was developed RTS and CTS has become common flow control signals for non-packetised communication, and should have been handled by hardware in the CDC spec.

[Nominal Animal]

Re USB: do you see the RTS/CTS equivalents on Ethernet?

Let's say you [passive you, not PlainName] have a shelf full of RS-232 doohickeys you want to interface your new computers to, but those new computers don't have hardware RS-232 ports.  Your first idea is to use USB CDC/ACM, i.e. "USB serial".

This is where things break down.  The USB CDC 1.2 – USB Class Definitions for Communications Devices version 1.2 from 2007, CDC1.2_WMC1.1_012011.zip – does not define a way for a host and and USB device to agree how that USB to serial bridge device should deal with flow control: RTS/CTS, XON/XOFF.


A practical, simple example:

Let's say you write a Linux/Unix/BSD application that opens a tty device (like an USB CDC ACM), sets the properties via termios interface, and writes some simple Hello! message, then waits and displays a response (if one arrives within say five seconds).  We also prepare an USB CDC device that displays all USB packets (or just sniff the USB bus using Wireshark or a bus analyzer).  We run the application a number of times, and only vary the RTS/CTS and XON/XOFF -related termios settings.  What differences (aside from normal timing variance et cetera) do we see in the USB bus traffic?  None.

I bet Windows behaves the same.

That is, the problem with generic USB CDC drivers is that they cannot provide the full termios state to the USB device, only a partial one including baud rate, parity, stop bits (1/1.5/2), RTS, and DTR.  Similarly, the USB device cannot report the full termios state to the host, only the partial one plus the six optional bits I already mentioned (errors, RI, break, DSR, DCD).

So, when your application sets RTS/CTS flow control, and you use generic/standard USB CDC drivers, that flow control setting is ignored, because the kernel cannot convey the information to the device (nor the device to the kernel).

Could we fix that?

Sure!  Just not using a single USB CDC ACM endpoint and standard drivers.

I do have a dirt cheap, simple as frig solution of sorts in mind. Basically, the device exports two COM ports (USB CDC ACM, no drivers needed).  By default, the first one is for the serial data, and the second one is for configuring the first one (using a terminal).  Basically, you stick it in your computer, open a terminal to configure it, and then open the data port in your preferred application.  The configuration port could even report parity errors and such, "conversationally".  Simples – assuming Windows can handle one USB device having two CDC ACM endpoints.

[westfw]

RTS and CTS in RS-232 were designed to control a half-duplex connection,

This is definitely true.
RTS meant "I have data, turn the MODEM around", and the MODEM would respond  with CTS: "OK, you are now allowed to send data.  The CTS usage ("you can send data") sounds almost "flow control -like"), which presumably is what caused it to be re-used.

However "hardware flow control" was widely in use well before USB use became widespread, and the only reason for it not to have been included in the USB/CDC spec is hidebound pedantry and spite.  :-(
No wonder so many vendors provided their own USB/UART drivers, rather than using CDC.

But we're talking about Software Flow Control, aren't we?  There's less excuse for CDC not supported that!  Grr.

[PlainName]

Could we fix that?

Sure!  Just not using a single USB CDC ACM endpoint and standard drivers.

OK, job done

I'm failing to see why that is the fault of RTS/CTS, or how RTS/CTS contribute to the problem. Isn't it a USB consortium screwup?

And, AFAICS, this is only an issue because USB includes serial port fakery so that it's notionally plug'n'play. Nothing to stop someone defining and creating their own driver on top of plain USB. It would be no less difficult to implement in the USB-serial dongle (except you won't buy an off-the-shelf chip for it) but, if you want to fake a COM port on the host it would need a driver writing there.

[Nominal Animal]

but, if you want to fake a COM port on the host it would need a driver writing there.

That's the thing: if you limit yourself to USB CDC ACM, you don't.  It does not natively support controlling flow control on the USB-to-serial bridge USB device, but it does support arbitrary baud rates et cetera.  XON/XOFF flow control on top of USB is ineffective because of the relatively large buffers and packet transport used.

And that is all there is to the "death of XON/XOFF flow control" and analogously 'death' of RTS/CTS flow control.  The generic USB CDC ACM protocol we use does just not support the host telling the USB device anything about flow control configuration, only about baud rate, parity, stop bits, et cetera.  Using *two*, one for the data itself and the other for providing a serial console (terminal interface) for configuring the data one, is just a real simple way to do it.  Equally, one could use an USB HID endpoint instead, and standard RawHID or FF interfaces to communicate at a modest bandwidth (a few kilobytes per second), again without anything except the bog-standard generic drivers all USB keyboards and mice and such use.  No drivers to install.

The idea that each USB device needs a dedicated OS driver is an odd Windows anachronism that I really, really don't understand.
All the OS needs to understand is the device class, and that's it: there is absolutely no need for dedicated drivers with standard USB protocols.

[magic]

I took a quick glance at the CDC spec and its' all about stuff like "telephones", "modems", "ISDN", etc.

I would guess that if flow control was not standardized then maybe those things simply didn't need it, and if they didn't need it then USB had no reason to bother standardizing it.

[Karel]

Funny exercise on windows:

Connect a microcontroller/device to a windows pc using a usb/rs232 converter. Feel free to use the latest and
greatest converter using highspeed usb (480Mb), bigger buffers, etc. Also, use the fastest pc or workstation
available with whatever version of windows since XP.

Let the device send a contnuous stream of data at 115K2 baud to the pc that uses some protocol to check for lost
bytes/characters.
Use some kind of test application on the pc that reads the incoming data from the virtual comport and just checks
for lost bytes/characters.

Disable any flowcontrol.

Don't use the pc for any other tasks and make sure it's not updating in the background.

You'll probably notice that no characters will be lost.

Now, using the mouse, grab the downright corner of a window of some application and start to quickly and
continuously resizing that window. Notice what happens with your incoming data....
Spoiler: lots of bytes lost caused by buffer overruns.

Conclusion: windows gives approx. 100 times more priority to graphical updates then to interrupts from the
usb hostcontroller chip on the motherboard of the pc. Also, you cannot have reliable rs232 communication
on windows without using hardware flowcontrol ( rts/cts) when the baudrate is more than 9600 baud.

The same is applicable to Linux but much, much less severe than windows.

[PlainName]

Conclusion: windows gives approx. 100 times more priority to graphical updates then to interrupts from the
usb hostcontroller chip on the motherboard of the pc.

Yes, that's by design because if it were the other way around users would complain about laggy display and similar. Windows for servers has a setting to invert that (because they are typically headless) so that background tasks take priority over the UI.

[IanB]

Funny exercise on windows:

Connect a microcontroller/device to a windows pc using a usb/rs232 converter. Feel free to use the latest and
greatest converter using highspeed usb (480Mb), bigger buffers, etc. Also, use the fastest pc or workstation
available with whatever version of windows since XP.

Let the device send a contnuous stream of data at 115K2 baud to the pc that uses some protocol to check for lost
bytes/characters.
Use some kind of test application on the pc that reads the incoming data from the virtual comport and just checks
for lost bytes/characters.

Disable any flowcontrol.

Don't use the pc for any other tasks and make sure it's not updating in the background.

You'll probably notice that no characters will be lost.

Now, using the mouse, grab the downright corner of a window of some application and start to quickly and
continuously resizing that window. Notice what happens with your incoming data....
Spoiler: lots of bytes lost caused by buffer overruns.

Conclusion: windows gives approx. 100 times more priority to graphical updates then to interrupts from the
usb hostcontroller chip on the motherboard of the pc. Also, you cannot have reliable rs232 communication
on windows without using hardware flowcontrol ( rts/cts) when the baudrate is more than 9600 baud.

The same is applicable to Linux but much, much less severe than windows.

But Windows is not a real-time operating system, so giving priority to the GUI is to be expected.

However, what happens if the same experiment is tried, but first increasing the process priority of the test program from the task manager, even to real-time priority?

[coppice]

But Windows is not a real-time operating system, so giving priority to the GUI is to be expected.

Sure, Windows is not a real time system, but its priority scheme is a mess. You can get things so badly scheduled that they clog your machine so it takes minutes to kill the offending task. The GUI can make your media stutter. Media is a key real time aspect of a modern GUI that Windows handles very poorly.

[Nominal Animal]

The same is applicable to Linux but much, much less severe than windows.

I haven't experienced that on Linux at all.  My preferred test method is generating a Xorshift64* random number sequence with seed sent first by the Linux application, using only 32 high bits of each generated 64-bit number (because that is one of the rare ones that pass all BigCrush tests; not even Mersenne Twister does).  When testing MCU-to-host, the host-side application generates the same sequence and compares it to the one it receives from the MCU.

As an example, Teensy 4.0 can sustain over 200 Mbits/s or 25 Mbytes/s to Linux hosts using USB CDC ACM, "USB serial", when writing in blocks of 32 bytes or larger.  Depending on the machine and load it varies between 21 and 29 Mbytes/s, with 25 Mbytes/s typical and expected, lower end only seen on some slow SBCs under heavy load.  I've tested this for hours on end, and seen no lost data.

If you lose USB data packets, it is a bug, either on the device or the host, either hardware or in the USB stack implementation.  USB is fundamentally not supposed to lose packets at all, because of the ACK/NACK responses.  AFAIK, all models of Raspberry Pis are known to have this fault, and it is only mitigatable in software, not fully fixable at all.  I do not know of any other Linux SBC that has this fault.

At 119200 baud, you use less than 12 kbytes/s.  To starve an application of this bandwidth, your process priorities must be set up wrong (i.e. deliberately niced too much), even on slow SBCs.  Even an ATmega32u4 (as in Pro Micro clones) can sustain over 900 kbytes/s, almost 1 Mbyte/s, over USB CDC ACM, continuously.

Thus, no.  I definitely have not seen this on Linux at all, outside of Raspberry Pis which have buggy USB hardware, not even on Linux SBCs or minilaotops.  It is purely a Windows and Raspberry Pi issue in my opinion and experience.

[free_electron]

You may have to disable the hardware fifo in the usb<>serial. the xoff is hidden in the fifo buffer, so you don't see it until the fifo is purged. By that time a new packet is in the fifo... arrived while you were purging. A CDC transport(assuming that is what you are using) is still 512 bytes with each frame being 64 bytes.

[Nominal Animal]

You may have to disable the hardware fifo in the usb<>serial [for XON/XOFF flow control on the host to work].

Yes, but that will absolutely kill the bandwidth.  At FS (12 Mbit/s), you'll max out the USB bus at about 1000 bytes/second – one byte sent per send slot opportunity; at HS (480 Mbit/s), at about 8000 bytes/second.  Neither suffices for 115200 baud.

[magic]

Connect a microcontroller/device to a windows pc using a usb/rs232 converter.

As an example, Teensy 4.0 can sustain over 200 Mbits/s or 25 Mbytes/s to Linux hosts using USB CDC ACM.

Thus, no.  I definitely have not seen this on Linux at all, outside of Raspberry Pis which have buggy USB hardware, not even on Linux SBCs or minilaotops.

That's a slightly different thing, isn't it?

[Karel]

[
If you lose USB data packets, it is a bug, either on the device or the host, either hardware or in the USB stack implementation. 

Who is saying you loose usb datapackets???

The mechanism is this:

- the device sends a continuous stream of data to the host (a windows pc)
- windows gets busy with something "more important"
- the rx buffer of the usb hostcontroller gets full because the os is not reading fast enough from it
- the usb host controller does not accept new datapackets from the usb slave (actually, it will not honour requests from the slave to upload data)
- the tx buffer of the usb slave gets full (rs232/usb converter chip) because it cannot and will not send datapackets when the hosts rx buffer is full
- the device keeps on writing new bytes into the rs232/usb converter chip because cts/rts are either not connected or ignored
- the rs232/usb converter chip either ignores the new bytes for tx or overwites the old bytes in the tx (fifo)buffer, hence the lost bytes

[iMo]

Have you confirmed that the system sending data does not respond to XOFF at all?  Ie - set up a second terminal emulator and typed your XOFF manually?  You could consider sending XOFFs pre-emptively, like at intervals half of what you'd expect to fill up whatever buffering you do have...

Yes, I have tried everything. The XOFF characters are simply ignored by the PC side.

Hmm, as I wrote above, when I issued ctrl-Q/ctrl_S from the keyboard (Win10, two TeraTerms against each other via FT232 dongles) the other side (transmitting side) stopped/started even at 921k properly..
Also the files I transferred were ok (sent as "binary" half duplex), while I saw the xon/xoff chars coming from the receiving side.
PS: while watching the xon/xoff channel I saw on o'scope the "C8 and 88 hex" chars coming in irregular intervals (those are the xon/xoff chars in reverse bit order when looking at it from the start bit). Perhaps somebody with a better o'scope with large mem may track those xon/xoff chars during a longer transfer and see how the buffering actually works..

https://www.eevblog.com/forum/general-computing/the-death-of-xonxoff-flow-control/msg5584567/#msg5584567

[magic]

No, Karel demonstrated that disabling flow control on Windows pretty much guarantees data loss, which means that Windows fails to schedule the consumer process (or the USB driver) often enough to drain hardware RX queues before they overrun.

XON/XOFF flow control does work on USB adapters. I have just tested with two PL2303 connected to each other. I was able to transfer a 10MB text file through the connection without any error, ten times. Without flow control, it would often fail after a few megs.

To enable (on Linux):

Code: [Select]

stty -F /dev/ttyUSB0 ixon
This functionality appears to be implemented in the USB-UART chips themselves, so it should work with low latency. Not all drivers implement it, for example PL2303 and FTDI do, but CH431 does not.

[m k]

This whole implementation is pretty silly, it's like updating display with a CPU.

Maybe the real problem was that Microsoft is a software company.
They did a legacy free PC, so obviously all *external* legacy *hardware* was also off.
The goal was that USB is the bridge, and it was, but other end had legacy software.
So obviously rest of the world had to react, I can still remember first CLI/STI tests, only few hundreds per second.

So legacy hardware was connected to legacy software through USB and obviously old stuff was tried to keep in the play.

Back then Microsoft had power and authorities didn't really understand.
When W95 came Microsoft cheated and changed things just before release.
So all external application software makers were lagging.
Before that Lotus 1-2-3 and WordPerfect were major players.

Hardware side was a bit different, but not much.
All industrial machines had to maintain their connections.
Parallel side was easy, just stream those same bytes over the USB.
Serial could have been the same, who messed it?

My guess is that the reason is Microsoft again.
They excluded device drivers from free compilers.
So big part of driver updaters were kicked out.

Back then parallel was already leaving behind, but serial was all around.
Changing some old external parallel devices was pretty easy when changing serial ones was a practical impossibility.
Parallel was beside you and serial somewhere else, and possibly very exotic, but also very vital.
An example, punched tape, YeeYee port connected punching machine was already left behind and direct serial added.

Next obstacle was SMB 1.0 but that's a different story.

[nctnico]

Have you confirmed that the system sending data does not respond to XOFF at all?  Ie - set up a second terminal emulator and typed your XOFF manually?  You could consider sending XOFFs pre-emptively, like at intervals half of what you'd expect to fill up whatever buffering you do have...

Yes, I have tried everything. The XOFF characters are simply ignored by the PC side.

Hmm, as I wrote above, when I issued ctrl-Q/ctrl_S from the keyboard (Win10, two TeraTerms against each other via FT232 dongles) the other side (transmitting side) stopped/started even at 921k properly..
Also the files I transferred were ok (sent as "binary" half duplex), while I saw the xon/xoff chars coming from the receiving side.
PS: while watching the xon/xoff channel I saw on o'scope the "C8 and 88 hex" chars coming in irregular intervals (those are the xon/xoff chars in reverse bit order when looking at it from the start bit). Perhaps somebody with a better o'scope with large mem may track those xon/xoff chars during a longer transfer and see how the buffering actually works..

The problem is that the USB adapters need to implement flow control in their drivers which is a hit &miss. And as Nominal Animal has researched, in USB-CDC UART flow control is broken even worse. The basically leads to a situation where UART flow control is highly hardware & software dependant which in turns means you can't rely on flow control to work using whatever hardware you have, nowadays. In your case you are just lucky it worked for you.

[magic]

It's more hit than miss unless your hardware is garbage, but software setup is tricky.

You say you tried with minicom. I found that this thing disables flow control (of all kind) when it starts and I'm not sure if that can be overriden in any way. But you can turn it back on after starting minicom, and then it works.

But yeah, probably not something you would want random customers to do.

edit
The problem of CDC limitations is resolved very easily: no one seems to make USB-RS232 cables based on CDC. I have only ever seen it used in devices which get data straight from USB packets and only pretend to be behind a serial port. CDC appears to be designed specifically for such devices and flow control built into USB bulk endpoints is all they need. Actual RS232 bridges use proprietary protocols.

[free_electron]

The problem is that the USB adapters need to implement flow control in their drivers which is a hit &miss.

it's a hardware issue. if the hardware could detect xon/xoff this problem would not exist.
- You are trying to run a protocol on top of a protocol on top of a protocol. ( xon on top of uart on top of usb ).
- add to that you are stuffing packets in packates (packet delineated by xon xoff on usb packets in frames.
- the usb hardware adds latency and double buffering. if a single character is received uart side it needs to be stuffed ina  64 byte usb frame.

this results in the xon/xoff characters to be somewhere in the stream.. by the time you receive them it is too late : there is more stuff already on the way. the same goes for outgoing xon xoff.. it is completely asynchronous with an unknown latency.
The only option is to make sure you have a buffer large enough to hold at least two complete usb frames (512 bytes). if the first frame contains the xon xoff you can react and there possible is another 512 on the way. taking into account the outgoing latency i would make it 3 frames. you may be sending while a second packet is in flight so it will take another 512 bytes at the opposite side to guarantee them trapping it and stopping transfer.

that is your only option. large buffers. or downprogram the buffer size in the uart. the ftdi devices have an early flag you can set on one of the pins that tell you the buffer has reached a threshold. you can tie that to an interrupt on the micro. ping the micro that the buffer is 1/3 or 1/4 full and the micro can send the xoff. anything in flight will still land in the built-in buffer

[nctnico]

The problem is that the USB adapters need to implement flow control in their drivers which is a hit &miss.

it's a hardware issue. if the hardware could detect xon/xoff this problem would not exist.
- You are trying to run a protocol on top of a protocol on top of a protocol. ( xon on top of uart on top of usb ).
- add to that you are stuffing packets in packates (packet delineated by xon xoff on usb packets in frames.
- the usb hardware adds latency and double buffering. if a single character is received uart side it needs to be stuffed ina  64 byte usb frame.

this results in the xon/xoff characters to be somewhere in the stream.. by the time you receive them it is too late : there is more stuff already on the way. the same goes for outgoing xon xoff.. it is completely asynchronous with an unknown latency.
The only option is to make sure you have a buffer large enough to hold at least two complete usb frames (512 bytes). if the first frame contains the xon xoff you can react and there possible is another 512 on the way. taking into account the outgoing latency i would make it 3 frames. you may be sending while a second packet is in flight so it will take another 512 bytes at the opposite side to guarantee them trapping it and stopping transfer.

that is your only option. large buffers. or downprogram the buffer size in the uart. the ftdi devices have an early flag you can set on one of the pins that tell you the buffer has reached a threshold. you can tie that to an interrupt on the micro. ping the micro that the buffer is 1/3 or 1/4 full and the micro can send the xoff. anything in flight will still land in the built-in buffer

No. As I wrote before, I tried all this as I have full control over the device which is receiving the data. It is the PC side which isn't dealing with xon/xoff properly and that can't be fixed in a way it will always work.

[Andy Chee]

Ben Eater released a new video, adding hardware flow control to his breadboard 6502.  No mention of USB issues, though perhaps his 6502 isn't running particularly fast baud rates.

[Nominal Animal]

The mechanism is this:

Ah, now I understand.  No, the rx buffer of the USB host controller does not get full (increasing it won't help), it is just that the host will not ACK the packet before the USB interrupt has occurred, so the device runs out of internal buffer.

Your complaint is that in Windows and in Linux that interrupt has too much latency for typical RS-232/UART to USB converters.

At 115200 baud, the data rate is about 11520 bytes per second.  If the converter to-host buffer size is N bytes beyond one USB packet –– remember, the converter cannot discard the sent USB packet until it has been ACKed, or indeed data may/will be lost ––, the allowed interrupt latency is N/11520 seconds, or N·86.8µs.  Let's say a typical converter has 128 bytes of buffer (so 64 bytes over one full packet at FS), which corresponds to 5.6ms.  Essentially, it is double-buffering the device-to-host USB packets.

Typical desktop systems often have longer interrupt latencies than that 5.6ms under load, because the scheduling unit is of that order.

In Linux, you can mitigate this by using a kernel configured to use 1000 Hz ticks (CONFIG_HZ=1000, CONFIG_HZ_1000=y).  Debian desktop (and derivatives like Ubuntu and Mint) defaults to 250 Hz, so the scheduling "unit" is 4ms.

The question is, how much buffer space is reasonable on any USB device?  That is, what kind of USB packet ACK latencies one should expect in practice?

In my opinion, triple buffering (or 192 bytes at USB 2.0 FS, 1536 bytes at USB 2.0 HS) should be the absolute minimum for any kind of pipe/bus with receive acknowledge.  I typically count on at least 4 USB packets: 256 bytes for FS, 2048 for HS.  More if available, of course.

This means that in my opinion, a "typical" 128-byte buffer is unrealistically small.  Especially when they claim to support 1 Mbaud (or higher!) rates, which corresponds (with a 128-byte buffer) to 0.56ms maximum USB packet-acknowledge interrupt latency.

Feel free to choose which one you blame: the OS for having USB packet ACK latencies longer than a couple of milliseconds, or the converter devices for having ridiculously small buffers for what they claim to do.  There are very valid reasons for both.

[Nominal Animal]

Do note that USB to serial converter chips that use their own driver instead of USB CDC ACM can properly support RTS/CTS and XON/XOFF flow control (on the converter chip itself).  It is only the USB CDC ACM standard/class driver that cannot.

[IanB]

Ben Eater released a new video, adding hardware flow control to his breadboard 6502.  No mention of USB issues, though perhaps his 6502 isn't running particularly fast baud rates.

I watched that video.

He was trying to paste large amounts of text into the 6502-based computer from a terminal program on the PC.

He used the hardware CTS/RTR pin on the microcomputer side to tell the PC to stop sending when the micro's UART buffer was nearly full, and resume when the UART buffer had more space. He set the terminal program on the PC to use RTS/CTS flow control.

It worked fine to avoid lost data. I think the success of this might have depended on what USB/RS232 adapter the PC was using. I have not watched the earlier videos to find that out.

[magic]

It works.

Shown below, blue is TX sending a stream of zero bytes and yellow is RX waiting for XOFF. As soon as that is received, the UART finishes the current byte and stops transmitting. The whole process takes nanoseconds, which certainly means that a round trip over USB to the CPU and back is not involved. This is PL2303, probably not even a genuine one.

FTDI would likely still send one more byte, because that's what it does when CTS/RTS flow control is used. This makes FTDI unusable with primitive MCUs with one byte RX buffer.

[magic]

The mechanism is this:

Ah, now I understand.  No, the rx buffer of the USB host controller does not get full (increasing it won't help), it is just that the host will not ACK the packet before the USB interrupt has occurred, so the device runs out of internal buffer.

Your complaint is that in Windows and in Linux that interrupt has too much latency for typical RS-232/UART to USB converters.

Of course a USB HC will ACK packets without CPU involvement, that's its damn job. How software can screw up is by not queuing enough transfer buffers - when the HC runs out of those, it will stop polling the endpoint for new data until more buffers are submitted by software.

[westfw]

no one seems to make USB-RS232 cables based on CDC.

It has become a thing.  People don't like installing "drivers" from "foreign" websites, that may or may not end up breaking existing setups (FTDI-gate?)  They would rather the dongle "just work with the standard drivers included with windows."

WCH, and I think SILabs or Cypress, have introduced new USB/Serial chips that have CDC/ACM support instead of custom drivers.  (eg https://www.wch-ic.com/products/CH9102.html )

[nctnico]

It works.

Shown below, blue is TX sending a stream of zero bytes and yellow is RX waiting for XOFF. As soon as that is received, the UART finishes the current byte and stops transmitting. The whole process takes nanoseconds, which certainly means that a round trip over USB to the CPU and back is not involved. This is PL2303, probably not even a genuine one.

FTDI would likely still send one more byte, because that's what it does when CTS/RTS flow control is used. This makes FTDI unusable with primitive MCUs with one byte RX buffer.

I don't think so. On such MCUs you'll have an interrupt + software ring buffer to receive data. If you use flow control, you set the high-watermark at a certain fill level of the ring buffer (say somewhere between 50% and 90%) which means you can still accept some extra data.

[PlainName]

It works.

Shown below, blue is TX sending a stream of zero bytes and yellow is RX waiting for XOFF. As soon as that is received, the UART finishes the current byte and stops transmitting. The whole process takes nanoseconds, which certainly means that a round trip over USB to the CPU and back is not involved. This is PL2303, probably not even a genuine one.

FTDI would likely still send one more byte, because that's what it does when CTS/RTS flow control is used. This makes FTDI unusable with primitive MCUs with one byte RX buffer.

I don't think so. On such MCUs you'll have an interrupt + software ring buffer to receive data. If you use flow control, you set the high-watermark at a certain fill level of the ring buffer (say somewhere between 50% and 90%) which means you can still accept some extra data.

I think you missed where he said it is the UART that sees Xoff and stops sending. Thus, at most, there would be one byte delay (the one already being sent).

The UART is the sensible place to do Xon/Xoff, but since it is in-band and you really don't want UARTs falling over on binary data, many if not most implementations do it in software and have the buffer issues. And, I might note, you also missed the sending buffer that either has to empty or get skipped before the Xoff even leaves the serial port. After all, Xon/Xoff is simply serial flow control using in-band data where there aren't enough signals, and it really wants to work at the same protocol level as the hardware signals it's replaced.

[Nominal Animal]

Of course a USB HC will ACK packets without CPU involvement, that's its damn job. How software can screw up is by not queuing enough transfer buffers - when the HC runs out of those, it will stop polling the endpoint for new data until more buffers are submitted by software.

Right.  The end result is exactly the same: USB packets not ACK'ed (or not enough transfer buffers queued, so the device has no opportunities to send more data), because the target application is not getting scheduled every few milliseconds; every half a millisecond at 1 Mbaud.

The solution used in Linux for USB SS –– for example, in SDR –– is to use async USB bulk transfers (via USBFS; not a file system, but the raw USB character devices at /dev/bus/usb/BBB/NNN) and queue a lot of them; and to increase the default USB buffer size from 64k (often up to 1G).

My point is that it is not reasonable for the device manufacturers to expect desktop OS applications to be able to always respond within milliseconds, because they never have. The other option is to say that OSes are buggy, because they cannot guarantee such low latencies.  Both are valid viewpoints, as there are reasons for and against for both.

It works. [...] This is PL2303, probably not even a genuine one.

Thing is, PL2303 uses its own driver and not USB CDC ACM, and it definitely supports both RTS/CTS and XON/XOFF flow control in the device.  (It only supports XON = 021/DC1/Ctrl-Q and XOFF = 023/DC3/Ctrl-S.)

That is, they implement flow control on the USB device itself, and the OS driver just tells it the flow control model to use.

PL2303G (aka type TXN) is extremely close to what a fixed USB CDC ACM would be.  It does have to use two Vendor-specific commands –– one for reading and one for writing the extra termios state –– and extends the SerialState notification message by adding bit 7, CTS.

To keep it short, if USB Class definitions for Communication Devices 1.2, and specifically PSTN120.pdf in it, were to

• Table 31: UART State Bitmap Values (page 33): add
  $$\begin{array}{l|l|l}
  \text{D7} & \text{bClearToSend} & \text{This signal corresponds to RS-232 signal CTS.} \\
  \end{array}$$
• Table 13: Class-Specific Request Codes for PSTN Subclasses (page 20): add
  $$\begin{array}{l|l}
  \text{SET_FLOW_CONTROL} & \text{24h} \\
  \text{GET_FLOW_CONTROL} & \text{25h} \\
  \end{array}$$
• Insert two new chapters between 6.3.13 and 6.3.14: SetFlowControl and GetFlowControl.
  bmRequestType=00100001B, wLength=0/2/4, Data=XON and XOFF codes (5/6/7/8/16-bit), and wValue is the bit mask of supported flow control modes.  I'd use bit 0 for RTS/CTS flow control, and bit 1 for XON/XOFF flow control, because I've heard of one strange device from a friend in Canada that needed both to actually work.  If wLength=0, XON/XOFF codes are fixed at 021/DC1/Ctrl-Q and XOFF = 023/DC3/Ctrl-S.  If wLength=2, the device supports arbitrary 8-bit XON/XOFF codes.  If wLength=4, the device supports arbitrary 16-bit XON/XOFF codes when Line Coding bDataBits==16.

then CDC ACM could support flow control just like PL2303 do.

Interestingly, the Linux PL2303 driver was written by Greg K-H, not Prolific; and in addition to the about twenty Prolific vendor:product USB IDs it supports, it also supports about fifty other ones too.  The PL2303G (aka TYPE_HXN) is very close to USB CDC ACM, and doesn't require the vendor message tickling (see drivers/usb/serial/pl2303.c:pl2303_startup(), and the vendor-specific messages non-HXN PL2303's require to work).  It only supports one port per device, though.

[magic]

FTDI would likely still send one more byte, because that's what it does when CTS/RTS flow control is used. This makes FTDI unusable with primitive MCUs with one byte RX buffer.

I don't think so. On such MCUs you'll have an interrupt + software ring buffer to receive data. If you use flow control, you set the high-watermark at a certain fill level of the ring buffer (say somewhere between 50% and 90%) which means you can still accept some extra data.

Fair enough, it will work if this is what you do and if you are always fast enough to read one received byte before another two bytes overrun the UART peripheral's RX queue.

But if you rely on the UART automatically driving flow control for you, you need the transmitter to stop after the current byte when it sees CTS transition in the middle of it. Been there, done that.

[magic]

Of course a USB HC will ACK packets without CPU involvement, that's its damn job. How software can screw up is by not queuing enough transfer buffers - when the HC runs out of those, it will stop polling the endpoint for new data until more buffers are submitted by software.

Right.  The end result is exactly the same: USB packets not ACK'ed (or not enough transfer buffers queued, so the device has no opportunities to send more data), because the target application is not getting scheduled every few milliseconds; every half a millisecond at 1 Mbaud.

The solution used in Linux for USB SS –– for example, in SDR –– is to use async USB bulk transfers (via USBFS; not a file system, but the raw USB character devices at /dev/bus/usb/BBB/NNN) and queue a lot of them; and to increase the default USB buffer size from 64k (often up to 1G).

My point is that it is not reasonable for the device manufacturers to expect desktop OS applications to be able to always respond within milliseconds, because they never have. The other option is to say that OSes are buggy, because they cannot guarantee such low latencies.  Both are valid viewpoints, as there are reasons for and against for both.

USB has always been designed to be cheap. USB devices are meant to be cheap and they are not expected to buffer more than a few packets worth of data, certainly not more than a (micro)frame worth. Hosts are supposed to poll them often enough for that not to be an issue. Host controllers are cheap too and they use system RAM for transfer buffers, and they have the important property that they accept packet queues in the form of linked lists of arbitrary length (not entirely true for periodic transfers before XHCI, but you could still queue many milliseconds ahead). It's the software's job to queue enough data buffers to cover its latency.

This is perfectly doable and hardly unique to SDR. When you play or record audio with a USB soundcard, you have many milliseconds worth of packets queued and you don't hear glitches when you resize a GUI window. Hell, even Windows probably gets it right or users would be whining loudly. There is no reason serial couldn't do it too.

It works. [...] This is PL2303, probably not even a genuine one.

Thing is, PL2303 uses its own driver and not USB CDC ACM, and it definitely supports both RTS/CTS and XON/XOFF flow control in the device.  (It only supports XON = 021/DC1/Ctrl-Q and XOFF = 023/DC3/Ctrl-S.)

That is, they implement flow control on the USB device itself, and the OS driver just tells it the flow control model to use.

Yes, that's my whole point. And a quick glance at Linux drivers suggests that most chips have this functionality in hardware. Notably excluding CH431, although it might also be that the HW has the ability but it's not implemented in Linux.

[guenthert]

The mechanism is this:

[..]
Your complaint is that in Windows and in Linux that interrupt has too much latency for typical RS-232/UART to USB converters.

At 115200 baud, the data rate is about 11520 bytes per second.  If the converter to-host buffer size is N bytes beyond one USB packet –– remember, the converter cannot discard the sent USB packet until it has been ACKed, or indeed data may/will be lost ––, the allowed interrupt latency is N/11520 seconds, or N·86.8µs.  Let's say a typical converter has 128 bytes of buffer (so 64 bytes over one full packet at FS), which corresponds to 5.6ms.  Essentially, it is double-buffering the device-to-host USB packets.

Typical desktop systems often have longer interrupt latencies than that 5.6ms under load, because the scheduling unit is of that order.

In Linux, you can mitigate this by using a kernel configured to use 1000 Hz ticks (CONFIG_HZ=1000, CONFIG_HZ_1000=y).  Debian desktop (and derivatives like Ubuntu and Mint) defaults to 250 Hz, so the scheduling "unit" is 4ms.
[..]

    Sorry, that's nonsense.  You seem to confuse task scheduling with interrupt handling.  Interrupt latencies in Linux on a modern desktop will be in the order of 0.01ms.   There is plenty of information about that out there, have a look at e.g. https://www4.cs.fau.de/Publications/2018/herzog_18_sbesc.pdf

[Karel]

The mechanism is this:

[..]
Your complaint is that in Windows and in Linux that interrupt has too much latency for typical RS-232/UART to USB converters.

At 115200 baud, the data rate is about 11520 bytes per second.  If the converter to-host buffer size is N bytes beyond one USB packet –– remember, the converter cannot discard the sent USB packet until it has been ACKed, or indeed data may/will be lost ––, the allowed interrupt latency is N/11520 seconds, or N·86.8µs.  Let's say a typical converter has 128 bytes of buffer (so 64 bytes over one full packet at FS), which corresponds to 5.6ms.  Essentially, it is double-buffering the device-to-host USB packets.

Typical desktop systems often have longer interrupt latencies than that 5.6ms under load, because the scheduling unit is of that order.

In Linux, you can mitigate this by using a kernel configured to use 1000 Hz ticks (CONFIG_HZ=1000, CONFIG_HZ_1000=y).  Debian desktop (and derivatives like Ubuntu and Mint) defaults to 250 Hz, so the scheduling "unit" is 4ms.
[..]

    Sorry, that's nonsense.  You seem to confuse task scheduling with interrupt handling.  Interrupt latencies in Linux on a modern desktop will be in the order of 0.01ms.   There is plenty of information about that out there, have a look at e.g. https://www4.cs.fau.de/Publications/2018/herzog_18_sbesc.pdf

Pay attention, both windows and Linux don't support nested interrupts! (at least not out of the box)
That means that, when a new interrupt arrives while the os is still servicing another interrupt, the new interupt will be serviced only after the other one has finished and, depending on the
source of the interrupt and how well the driver for that interrupt has been written, that can take a considerable amount of time.
And no, interrupts are not spreaded over multiple cores, so no multitasking there...

[m k]

I have full control over the device which is receiving the data. It is the PC side which isn't dealing with xon/xoff properly and that can't be fixed in a way it will always work.

So you can trig when start bit happens?
Send Xoff then.

You'll still receive what is in the furthest buffer, so that you must handle.
After that you can send Xon.
If your Xoff wont go through in time you need more buffer.

If your Xoff is dealt by UART, no problem.
If USB, who knows yes, but if it can't receive your Xoff it can't send anything either, if it's not broken and flow control is on.

Maybe I didn't understand something.
You changed to xmodem protocol, so both ends must do something, since xmodem is not a default thing.
Similarly both ends can switch software flow control on.
Or are you saying that PC is not obeying your flow control setting at all?

software flow control test
copy file com1: to copy com1: con:
W10 -> W95 OSR2, no.
W95 -> W10, no.

W95 mode doesn't have flow control settings.

W95 HyperTerminal to W10 puTTY is fine, Xoff is supported.
W10 copy file com1: to W95 HyperTerminal is fine, Xoff is supported.

W95 HyperTerminal to W10 copy com1: con: is unknown, Xoff is not sent.
So keyboard is not con: anymore or I can't see the 1200 baud LED change.

After puTTY W10 COM settings are what puTTY defined.
Console mode baud alone defaults parity and data bits, but not flow control.

W95 is different, its DOSbox is not a box.

So native W10 clearly supports software flow control.

XP SP2 is like W10.
So it also supports software flow control natively.

Vista machine has no serial port.
Maybe that's a good thing.

[nctnico]

I have full control over the device which is receiving the data. It is the PC side which isn't dealing with xon/xoff properly and that can't be fixed in a way it will always work.

So you can trig when start bit happens?
Send Xoff then.

You'll still receive what is in the furthest buffer, so that you must handle.
After that you can send Xon.
If your Xoff wont go through in time you need more buffer.

If your Xoff is dealt by UART, no problem.
If USB, who knows yes, but if it can't receive your Xoff it can't send anything either, if it's not broken and flow control is on.

Maybe I didn't understand something.
You changed to xmodem protocol, so both ends must do something, since xmodem is not a default thing.
Similarly both ends can switch software flow control on.
Or are you saying that PC is not obeying your flow control setting at all?

Exactly. And the problem lies somewhere between the API I/O layer and the USB dongle doing the USB-to-UART conversion. A cursory look into the Linux kernel source shows that implementing flow control is handed over to the driver to deal with it which is not always the case (or may have gotten broken in the many Linux kernel refactoring cycles). So I decided to use a transfer method which is handled at the application level entirely and thus bypassing any possible brokeness in OS layers (including device drivers) and hardware. I have spend a couple of days on this problem and at some point I needed something which just works.

[Nominal Animal]

My point is that it is not reasonable for the device manufacturers to expect desktop OS applications to be able to always respond within milliseconds, because they never have. The other option is to say that OSes are buggy, because they cannot guarantee such low latencies.  Both are valid viewpoints, as there are reasons for and against for both.

USB has always been designed to be cheap. USB devices are meant to be cheap and they are not expected to buffer more than a few packets worth of data, certainly not more than a (micro)frame worth. Hosts are supposed to poll them often enough for that not to be an issue. Host controllers are cheap too and they use system RAM for transfer buffers, and they have the important property that they accept packet queues in the form of linked lists of arbitrary length (not entirely true for periodic transfers before XHCI, but you could still queue many milliseconds ahead). It's the software's job to queue enough data buffers to cover its latency.

This is perfectly doable and hardly unique to SDR. When you play or record audio with a USB soundcard, you have many milliseconds worth of packets queued and you don't hear glitches when you resize a GUI window. Hell, even Windows probably gets it right or users would be whining loudly. There is no reason serial couldn't do it too.

Perhaps.  These USB devices are also prone to stop working (or being yanked out) at any moment, and it is even more important to not lock up (even temporarily) when that happens, or gobble up unlimited amounts of RAM (especially on routers and appliances with very limited amounts available).  You can see this all over the Linux USB device drivers, what with timeouts and automatic backoffs (1ms backoff/latency added whenever USB device NAKs a packet).
Thus, I'm hesitant to fully agree.  I'd need to see/hear it done right on a desktop OS first, to believe it can be done significantly better.

The wrench in the cogs in Linux/Android/Mac/BSDs is the tty layer.  It is traditionally capped at 4096 bytes of buffer, and just isn't designed for high throughput.  (You can observe this by creating a simple pseudoterminal pair, and push data through locally.  The bandwidth you can achieve is surprisingly low, compared to e.g. a socket pair or a pipe.)  When using USB 2.0 HS (480 Mbit/s) microcontrollers, it becomes a bottleneck.

Because of Teensy 4, I've played with a number of possible solutions, suitable for CDC ACM (and close enough converter chips like some PL2303 variants).  The bulk data is trivial via a simple character device driver, and configuration via ioctls (even reusing the termios tty ones).  Even a pair of vendor commands could be reasonably implemented via ioctls, with just the actual USB message format varied depending on vendor:product or quirk.  Notifications are what  has given me a bit of a pause, because you can easily get 8000 of them per second.  POSIX signals are an obvious choice, with dedicated ioctl receiving the off-band message, but considering the possible frequency, it isn't optimal.  Obtaining a secondary descriptor for notifications would be better, but this is unusual, so proper destruction especially when the bulk one is closed first gets a bit complicated.  And then there are more abstract issues, like whether packet boundaries should be respected with read(): yes allows for datagram approaches, no is better for bulk bandwidth.  Linux input subsystem (using whole struct input_event datagrams only) shows how useful the datagram approach is, being rock solid for a quarter of a century now, and yet easily extended.  And we do have both datagram and stream local/UNIX domain sockets for a reason.

Sorry, that's nonsense.  You seem to confuse task scheduling with interrupt handling.

You seem to forget the effect of the tty layer and softirq handling in between the userspace process and the USB device.

Because this is not a Linux kernel developer forum, I'm not trying to be technically correct:  I'm trying to describe the observable behaviour of the system, not its implementation details.  Just pointing out I'm technically wrong is not helping anyone: at minimum, you should provide the correct description.

In particular, see for yourself the difference between CONFIG_HZ=250 and CONFIG_HZ=1000, specifically the difference in latency from the USB device point of view when using USB serial (CDC ACM or vendor driver, doesn't matter).  If we take your post as the whole truth, then there would be no difference, would there?

I do err quite often, and I'm ready to acknowledge my errors.  Fact is, if we ignore the liberties I've taken with the technical descriptions, I still stand by the cause-and-effect chain/interactions I've tried to explain here.  Whether it is the actual interrupt latency, or the latency at which the softirq task in the tty layer manages to push data forwards allowing further transfers to occur, is simply irrelevant to me at the level of complexity/abstraction we're discussing here.  If you can describe it better, please do, and don't just strike down others who try.

[Karel]

The wrench in the cogs in Linux/Android/Mac/BSDs is the tty layer.  It is traditionally capped at 4096 bytes of buffer, and just isn't designed for high throughput.  (You can observe this by creating a simple pseudoterminal pair, and push data through locally.  The bandwidth you can achieve is surprisingly low, compared to e.g. a socket pair or a pipe.)  When using USB 2.0 HS (480 Mbit/s) microcontrollers, it becomes a bottleneck.

The "real" tty driver uses a flipbuffer of 4096 bytes (2 x 4096 bytes). When one is full (or a timeout occurred), it's handed
over to the applicationn layer and it starts filling up the other buffer (it flips the buffer).

The "virtual/usb" tty driver uses the same mechanism but instead of a flipbuffer of 2x 4096 bytes, it uses
10 buffers of 4096 bytes (in a fifo setup). This way, the throughput of the old tty driver is enhanced a lot.

[Andy Chee]

Exactly. And the problem lies somewhere between the API I/O layer and the USB dongle doing the USB-to-UART conversion.

In four pages of discussion, you have yet to mention what baud rate you are running at.

Does your Xon/Xoff problem exist at all baud rates?

[magic]

It exists at all baud rates fast enough to fill the target device's RX buffer in whatever time it takes to wait for the preceding flash write to complete, obviously

The OP's problem is that flow control apparently wasn't even enabled in his tests in the first place. And even if it were, a second problem exists that not all HW supports it.

[Andy Chee]

And even if it were, a second problem exists that not all HW supports it.

This seems ripe for a benchmarking experiment across multiple OS platforms (Win 98, Win XP, Win 7, macOS, Linux) and multiple USB-serial dongles (FTDI, PL2303, CP2102).

So what's needed is a standardised "black box" which can send/receive xon/xoff bytes, maybe a PIC micro LCD terminal with a 16 byte receive buffer, send Xoff when the buffer reaches 12 bytes full, send Xon when the buffer reaches 4 bytes full?

[nctnico]

It exists at all baud rates fast enough to fill the target device's RX buffer in whatever time it takes to wait for the preceding flash write to complete, obviously

The OP's problem is that flow control apparently wasn't even enabled in his tests in the first place. And even if it were, a second problem exists that not all HW supports it.

Flow control was enabled. Just not handled the way it should be.

[magic]

OK, cool.

Neither Linux and Windows have working xon/xoff handshaking implemented. More precisely, the xon/xoff function is handed to the driver and (next to) none of the hardware manufacturers seems to have implemented this form of flow control.

So what sort of hardware have you tested and found not to implement flow control?

Because I have demonstrated that it clearly works as advertised on PL2303 dongles, and later also checked FTDI (as expected - one more byte sent, then it goes silent). I haven't tried built-in motherboard UARTs, but I would be highly surprised if they don't work. I haven't tried Windows either, but pretty sure the above vendors implement it too.

So far the only widespread UART that I know not to support it is CH341.

[Nominal Animal]

So far the only widespread UART that I know not to support it is CH341.

Actually, the Linux reverse-engineered ch341 driver does not (handle either RTS/CTS or XON/XOFF handshaking), but the WinChipHead CH341 Linux driver – it's provided as GPLv2 source code, not binaries  – has the code to support RTS/CTS handshaking on the device.  Windows users might wish to check the corresponding Windows driver, too, instead of using the built-in one.

I don't have any CH341-based converters myself to test and verify, though.  It could be it doesn't actually work in real life.  (Because of this, I didn't check the driver sources for anything suspicious, either.)

If that works, the support could be easily ported to the upstream kernel ch341 driver too, the licenses matching and all.  To enable RTS/CTS flow control, one needs to usb_control_msg(dev, pipe, 0x9A, USB_TYPE_VENDOR | USB_RECIP_DEVICE | USB_DIR_OUT, 0x2727, 0x0101, NULL, 0, DEFAULT_TIMEOUT); and to disable it, usb_control_msg(dev, pipe, 0x9A, USB_TYPE_VENDOR | USB_RECIP_DEVICE | USB_DIR_OUT, 0x2727, 0x0000, NULL, 0, DEFAULT_TIMEOUT);.  Both utterly normal USB vendor control messages.

So, it looks like for RTS/CTS, we're down to USB CDC ACM.  I suspect there are many that do not implement XON/XOFF; CH341 doesn't.

[magic]

https://www.spinics.net/lists/linux-serial/msg21945.html
https://bugzilla.kernel.org/show_bug.cgi?id=197109
https://lore.kernel.org/all/20240905224326.7787-1-me@lodewillems.com/T/
 

The vendor driver is GPL, but it uses magic numbers everywhere so probably it wouldn't be accepted upstream, and for some reason either WCH doesn't want to work on improving the existing driver, or some maintainer or another doesn't want their contributions for some reason, or they CBA to clean up their code. I don't know what's going on, but it looks like a case of problem(s) sitting between keyboard and monitor.

(In addition to the fairly anemic response to 3rd parties trying to fix the upstream driver, as shown above.)

[Nominal Animal]

I don't know what's going on, but it looks like a case of problem(s) sitting between keyboard and monitor.

Yep.  Note how I didn't say it would be easy to push those changes upstream.  Patches along the same lines for ch341 were first submitted in 2016, AFAICT.

The subsystem maintainer is Johan Hovold, and the proper list for this is linux-usb (MARC archives).  I suspect it would take someone willing to obtain a number of ch341 variants for verifying the changes work on them as well, revamping the driver to better conform to current USB driver style (there's a lot of unnecessary old baggage there), and work with the maintainer (Hovold, and Greg K-H for pushing the changes to -stable) at least medium-term to get it up to snuff.  Starting with following the proper patch submission procedure, especially the checklist.

(They're not grumpy, either one, but it takes a resilient person to bug the list once per week until responded, then work through every single complaint or suggestion.  It can be frustrating work for very little gain, unless someone does it to gain linux kernel developer experience, and the know-how on how to go about upstreaming stuff.  They will be blunt, and you will be ignored at first, so gentle but insistent perseverence and resilience is absolutely necessary.)

The reason WCH doesn't do this is because they don't have to.  Paying a Linux kernel dev to do this would not increase their business any.  I don't know if they have samples of different (incl. past) versions of ch341 chips for testing, but since they bothered to GPL their driver, they might help if it doesn't cost them anything.

For the devices I have, I prefer to use Teensy 4's (made into ad-hoc USB serial converters).  Most of the SBC and router consoles are RX+TX only, no RTS/CTS flow control, so I like to use the large buffers to catch everything even if I don't have an application open on the port.  And isolators sprinkled here and there, all willy-nilly.

[nctnico]

OK, cool.

Neither Linux and Windows have working xon/xoff handshaking implemented. More precisely, the xon/xoff function is handed to the driver and (next to) none of the hardware manufacturers seems to have implemented this form of flow control.

So what sort of hardware have you tested and found not to implement flow control?

I'm using CP20x USB-UART converters.

I recall some of the posts you linked to and while you can argue broken drivers shouldn't exist, the reason they exist is because people appearantly no longer care about having working flow control. This may sound like a big leap to you but I really like to be sure that whatever I send out into the field, works with 99.99% of hardware / software so there is no dissapointment from the user's side if that can be avoided. Better safe than sorry.

[magic]

The reason WCH doesn't do this is because they don't have to.  Paying a Linux kernel dev to do this would not increase their business any.  I don't know if they have samples of different (incl. past) versions of ch341 chips for testing, but since they bothered to GPL their driver, they might help if it doesn't cost them anything.

Well, they are maintaining this out of tree driver and adding new kernel version #ifdefs every now and then. So it already costs them something and there is apparently a business case for having a reasonably functional Linux driver.

So what sort of hardware have you tested and found not to implement flow control?

I'm using CP20x USB-UART converters.

Hmm, did you mean Silicon Labs CP210x?

Their datasheets advertise "hardware Xon/Xoff handshaking supported" and something of that sort appears to be implemented in the Linux driver. Theoretically, it may be buggy, but IMO there is at least an equally good chance that your software configuration was simply wrong, as it is my direct experience that getting it right is tricky.

I would test this myself right now if I had this hardware, but I don't, so I can't.

[iMo]

Here is an older FTDI pdf on Xon/Xoff..

[nctnico]

The reason WCH doesn't do this is because they don't have to.  Paying a Linux kernel dev to do this would not increase their business any.  I don't know if they have samples of different (incl. past) versions of ch341 chips for testing, but since they bothered to GPL their driver, they might help if it doesn't cost them anything.

Well, they are maintaining this out of tree driver and adding new kernel version #ifdefs every now and then. So it already costs them something and there is apparently a business case for having a reasonably functional Linux driver.

So what sort of hardware have you tested and found not to implement flow control?

I'm using CP20x USB-UART converters.

Hmm, did you mean Silicon Labs CP210x?

Their datasheets advertise "hardware Xon/Xoff handshaking supported" and something of that sort appears to be implemented in the Linux driver. Theoretically, it may be buggy, but IMO there is at least an equally good chance that your software configuration was simply wrong, as it is my direct experience that getting it right is tricky.

How can setting a program to use software flow control go wrong? Just set 'software flow control' to enabled; it shouldn't be any more difficult than that. Especially since I tested several different terminal emulators which all have the exact same problem. I can get that 1 program might fail due to lack of testing, but minicom and Putty are pretty solid in my book and those failed as well.

[Nominal Animal]

The reason WCH doesn't do this is because they don't have to.  Paying a Linux kernel dev to do this would not increase their business any.  I don't know if they have samples of different (incl. past) versions of ch341 chips for testing, but since they bothered to GPL their driver, they might help if it doesn't cost them anything.

Well, they are maintaining this out of tree driver and adding new kernel version #ifdefs every now and then. So it already costs them something and there is apparently a business case for having a reasonably functional Linux driver.

Implementing a working USB serial driver is easy and cheap: there are many already in the upstream kernel to look at as a guide.  It is finessing the driver into a form that is acceptable to upstream that is more expensive, because it requires someone to persevere and commit to the project until it is in acceptable shape and included – plus stay on afterwards, enough to respond to any bug reports and to add the new device IDs and such.

It is not that expensive, and there are already SoC vendors like RockChip who employ such developers because it makes financial sense.  In WinChipHead's case, they already have an in-tree driver (that does not support hardware flow control, and apparently has other issues as well, for example with break line condition) and their own out-of-tree GPL2 source driver, which makes it hard to justify any further investment.

Of course, if they had such a developer to support all their chips, with that developer having support from the company to do the job properly, upstreaming new device support and so on, they could change the perception of the company and their products among kernel developers and systems integrators (those building their own "distros" for appliances like routers and such).  I think it would be worth it within a year or two, but me and business planning aren't friends anymore, so I could easily be wrong.

In a way, it might be fun to do, actually.  The hardest part would be to obtain samples of the various versions of the chips, and actually testing the "fixed" driver works correctly over the whole known range, with some automated test benches or data flow tests.  That sort of hardware testing would also be very much appreciated by the subsystem maintainers, and make patch submission much easier; it looks like a major issue with past patch submissions was the inability to check whether they negatively affected other versions of the ch341 devices.

It would also be fun to find out how the USB Implementers Forum would react to the suggested additions for flow control configuration to USB CDC 1.2 spec.  I very much doubt they'd even read emails from a non-member, much less an open source developer.  (WCH = Nanjing Qinheng Microelectronics Co., Ltd., vendor 0x1A86.)  I don't think it'd succeed, but again, I'm often wrong.

[Nominal Animal]

How can setting a program to use software flow control go wrong?

The Linux CP210x driver does support XON/XOFF flow control, and with any XON and XOFF characters, too.  When you plug in the converter, could you check if
    stty -a -F /dev/ttyconverter
output contains start = ^Q; stop = ^S; ?  It should, but if not, it'd be the reason for the flow control issue.

If not, then in Linux, you need to set them:
    stty -F /dev/ttyconverter start '^Q' stop '^S' ixon ixoff
because obviously the terminal applications won't.  This is easiest to do in a suitable udev rule matching the device, and adding RUN:="/sbin/stty -F /dev/%k start '^Q' stop '^S' ixon ixoff" (or other suitable termios settings).

However, the driver mentions erratum CP2102N_E104, saying that CP2102N in QFN20, QFN24, QFN28 packages with firmware versions up to 0x10004 do not support flow control.  I hope your device is not a CP2102N?

[nctnico]

Output looks OK to me:

stty -a -F /dev/ttyUSB0
speed 115200 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W;
lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc

The chip used is a CP2103
With xon/xoff flow control enabled the xon/xoff characters are absorbed somewhere but they just don't have any effect.

I don't think you should rule out there is a bug in the specific kernel version I'm using even though it is a vanilla one from Debian. I've seen some weird crap where device drivers where partially rewritten to support new kernel data structures and others where not.

[magic]

This looks like it's supposed to work, at least on the transmitting side. But are you running this before or after starting the terminal emulator?

As I said, minicom simply disables all flow control when it starts and I found no way to change that without running stty again

[m k]

Found Belkin F5U103v 067B:2303
XP Pro SP3 with ser2pl.sys 2.1.51.238 is fine.

lubuntu 18.04.6 LTS 4.15.0-204-generic
stty -F /dev/ttyUSB0 1200 raw pass8 ixon
cp [file] /dev/ttyUSB0
No, Xoff sent, but transmit not stopped.
(stop = ^S)
'sane' setting didn't change anything.
Stuff moves to both directions, cp /dev/ttyUSB0 [file] is fine.

HP OEM Asus K8S-LA (SiS 760GX), puTTY not starting, can't open port.
Downgrading, can't remember the size of 's' of default puTTY serial.

[nctnico]

I just tried: if you turn software flow control on in minicom, it will configure the port to enable xon/xoff flow control. It is just that minicom doesn't restore the old stty values on exit.

[m k]

Downgraded to lubuntu 14.04.6 LTS 3.13.0-13-170-generic.
Same machine.

puTTY still nagging that ttyS0 is no good.
cp [file] /dev/ttyS0 is fine and Xoff supported.

Belkin 067B:2303 is still the same, no Xoff.

Packard Bell OEM MSI MS-6786 (VIA KM400A/KN400A)
lubuntu 14.04.6 LTS 3.13.0-13-170-generic and
lubuntu 10.04 LTS 2.16.32-21-generic
same thing, Belkin 067B:2303 no Xoff.
Standard port is fine.

[m k]

HP Maverick Silverado Rev 0B (Intel Q965)
lubuntu 16.04.6 LTS 4.4.0-185-generic
Belkin dongle still bad and standard port good.

My give up.
It's not the other hardware, these Prolific drivers just don't work.

Also found brainboxes VX-001B ExpressCard dated 17/8/11 and HP DV6 Vista has a port.
I think I'll pass.

[free_electron]

because people appearantly no longer care about having working flow control.

old protocol to circumvent latency (xon/xoff) over another old protocol (serial port) shoehorned on a packetized bus with more latency.

Why don't you make a proper USB device ? i know the answer... because then YOU have to provide all the drivers for all the platforms. (I face the same problem.)
We keep encountering uart style transport over usb because of the immense task of making USB drivers for a plethora of operating systems. (and the associated issues getting them certified and whcl approved so the OS will allow them to run)

Can you use a HID type device ? then you don't need drivers.

[nctnico]

A HID style device can't be used with a terminal emulator program. It may be old school but using a serial port and a terminal emulator takes away a lot of effort to create a way for development and debugging as it uses mature technology which is widely supported and available for any platform (including mobile phones)  You are right about not wanting to create custom drivers.

[PlainName]

Can you provide an Ethernet interface instead of serial? Then it would work with any network, as do most terminal apps, and use TCP flow control.

OK, a bit more onerous than a serial port but you wouldn't have to write any drivers, and it would work to practically anything (even wireless with an appropriate dongle).

[nctnico]

Can you provide an Ethernet interface instead of serial? Then it would work with any network, as do most terminal apps, and use TCP flow control.

No, because that would require configuring a network and thus a user interface (=serial port for example), having extra electronics, etc, etc. It is a complete non-starter. Bringing a serial port out is just a matter of extra test pads. The ethernet parts alone costs more than the entire BOM for some of my designs.

[Nominal Animal]

Do remember that it is only passing the full flow control settings from host to device that does not work with plain CDC ACM.

If your USB microcontroller is hard-wired to another microcontroller over UART, or only supports a fixed set of settings (baudrate, parity, stop bits, flow control), the USB microcontroller can simply ignore the serial port settings the host provides, and always use its own fixed settings.

[m k]

Maybe original idea was a partial connection.

Is raw text print available?

[m k]

Seems that the easiest thing is a pass through port and own sending script.

while (all not sent) {
 while (something received) {
  receive character
  update Xoff status
 }
 check Xoff status
 if (not Xoff)
  send character
}

[Nominal Animal]

For what it is worth, I blundered into a very interesting yet cheap way to fix the problems at the hardware level.

WCH CH32V203C8T6 is a Qingke V4B RISC-V MCU in LQFP48, which has two full-speed USB interfaces, but costs less than $1€ in singles at LCSC (and Aliexpress).  It does not require a crystal; a 3.3V regulator and two 100nF X7R supply bypass capacitors suffice.  However, it does have four UARTs, two SPI, and two I2C.  See Stefan Wagner's CH32V203 F6P6 development board as an example, one using the F6P6/TSSOP20 variant.

On the C8T6, one can use both USBs, I2C1 (SCL on 45/PB8 and SDA on 46/PB9, alternate configuration), USART2 and USART4 (RX, TX, CTS, RTS), SPI1 (NSS, SCK, MOSI, MISO), and either SPI2 (NSS, SCK, MOSI, MISO) and I2C2 (SDA, SCL), or USART3 (RX, TX, CTS, RTS), at the same time.

The idea is simple: you have a small device exposing aforementioned pins.  One USB device (USBD) is the bulk data interface.  The other USB device (USBFS) is an optional one, and provides a terminal console (using standard ANSI escape codes) where one can interactively configure and monitor the bulk data interface, for example using Minicom/PuTTY etc.  You don't need to use the console interface, unless you wish to configure the bulk interface, but the configuration interface could for example make certain things, like say baud rate, to some fixed value, having the bulk interface lie to the OS.  Useful in cases where you like a specific terminal software, but it has issues setting all the settings the way you like, or the driver does things you don't like.  I'd add an I2C EEPROM like M24C01-RMN6P or M24C08-RMN6TP for about ten cents, to store the configuration without wearing down the Flash on the MCU.  In all, I calculate the BOM cost to be less than 2 USD/EUR.

In Linux of course, one can do the same thing just using different CDC ACM endpoints and udev rules to identify them.  Here, the two USB devices can have different Vendor:Product and/or Serial, identifying which is which on all operating systems, even Windows, when still using the generic/class CDC ACM drivers.

Two separate USB devices also makes for interesting shenanigans, since each USB is limited to about 1 Mbytes/sec data throughput.  For example, for a custom bulk SPI stuff, one could use one USB for outgoing data, and the other for incoming data, for sustained continuous SPI transfers at about 8 MHz, effectively doubling the full-speed USB bandwidth.  Also, resetting/reconfiguring the bulk data interface for a different set of endpoints won't reset the configuration interface, because they are separate USB devices.

[magic]

With an MCU you can make your own USB-UART bridge which always obeys flow control regardless of host software settings. It could even implement CDC to work everywhere.

[westfw]

make your own USB-UART bridge ... It could even implement CDC]/quote]
You haven't been paying attention.  If you implement CDC, there's no way for the host-side software to turn flowcontrol on and off.
I guess that would be OK, if you're willing to have it turned on permanently, or via a HW jumper, or some magic UART sequence on the device side, but it's certainly not ideal.

(oooh... there's an interesting topic.  How and what commands would you implement on the UART side of such a device, if you want to also maintain full 8bit data transparency "most of the time."  BREAK comes to mind; Both the bridge (if a microcontroller) and target (probably also a microcontroller) should be able to create and measure timed BREAK conditions.)
Edit: likes like UARTS with LIN support may already do some of this - BREAK 0x55 is apparently supposed to help do autobaud.)

[Nominal Animal]

It's all about making it easier to set and override software settings for the hardware serial port.  USB CDC is wonderfully easy to use, but insufficient for setting flow control on the device itself.

BREAK condition is when the data line is asserted low continuously, for longer than a full symbol.  Following that with 0x55 gives you four one-bit wide high pulses immediately following the low.  This is sufficient for baud rate detection, as the timing is not that strict (±5%, although many prefer to get within ±1%).

For a continuous UART output, a timing histogram of the shortest bit durations can be used for baud rate detection.  This is slightly more complicated if one wants to be robust against glitches; plus the fact that stop bits can be 1, 1.5, or 2 bit times wide.

[Jeroen3]

I'm in a bit of a pickle with a project which needs to receive large amounts of text (more than anticipated) but it can't keep up with processing and buffering is not practical.

Maybe I'm kicking in an open door here, but if the baudrate is too high to keep up with on the mcu...
What if you lower it?

[magic]

Then the firmware upgrade takes longer
And you must be damn sure that programming a page will never take longer than expected, because RX buffer overrun would result.

By far the best option is to have more RAM than you have flash