Dealing with Serial FIFO Backlog

[SJL-Instruments]

We have a product with an STM32 MCU and standard USB serial interface.
The USB serial commands go into a FIFO buffer, and some commands can take a few seconds to complete (time taken is not predictable).
It's possible to queue up many of these commands, effectively locking up the serial interface until all commands complete.

A few possible ways to address this problem:
1. Ignore it, as the device is technically doing what was asked by the user. Not ideal.
2. Add a command that immediately cancels the current operation and clears the buffer. But, this would require the user to know about the existence of this command.
3. Require periodic "pings" from the host computer (every few seconds or so), and reset the MCU if not received. But, this would complicate the implementation for the end user, and make the device difficult to use from an interactive terminal.
4. Severely limit the size of the FIFO (to a few commands). Again, not ideal, and doesn't really fix the problem.

We're currently leaning towards solution 2.

Curious if anyone has a better solution, or if there's an industry-standard way to deal with this problem. Feels like we may be reinventing the wheel.

[bson]

5. Define your protocol to only accept a certain number of pipelined characters, and add a command to query what this is.  Then leave it to the host computer implementation to not exceed it.  Return an error (to help debugging) if it's exceeded.
6. Implement a command queue and only use the UART FIFO for what it's intended for - to reduce the number of interrupts.  But you still have to implement some sort of policy (if nothing else so the host computer doesn't keep pipelining commands until it has filled up memory, or a fixed sized footprint command queue).

I'd go with 5 if the FIFO is adequate, otherwise a command queue is needed (in memory).

[SJL-Instruments]

5. Define your protocol to only accept a certain number of pipelined characters, and add a command to query what this is.  Then leave it to the host computer implementation to not exceed it.  Return an error (to help debugging) if it's exceeded.
6. Implement a command queue and only use the UART FIFO for what it's intended for - to reduce the number of interrupts.  But you still have to implement some sort of policy (if nothing else so the host computer doesn't keep pipelining commands until it has filled up memory, or a fixed sized footprint command queue).

I'd go with 5 if the FIFO is adequate, otherwise a command queue is needed (in memory).

Thanks for the suggestions. To clarify, we have implemented a software command queue with 512 slots (this is what we referred to as the FIFO). The issue is not that the command queue is filling up (there's plenty of room), it's that some commands may require gathering data for up to 5 seconds.
If the user sends hundreds of these commands, the MCU will slowly work its way through each one, and any new commands won't be processed until the previous ones complete. Our official software doesn't back up the queue like this - it's mainly a potential problem for custom end-user scripts.

Technically, it's behaving exactly as intended. This might just be how most devices operate - we just wanted to check whether this is considered acceptable, or if there's a standard workaround.

[nctnico]

IMHO the protocol is designed wrong. Don't queue commands but work in a request - response style. That way the host and device are always in sync. This also helps to deal with errors and to re-synchronise the interface in case a command doesn't make it to the device somehow. If you are worried about dead time due to turnaround, design new commands which replace a series of commands and / or allow a command to return multiple results in a defined way.

I have dealt with & implemented dozens of different protocols from many vendors and designed my own as well over the decades and all of them use a request - response structure.

[Siwastaja]

+1 to request-response model, PLUS add unique transaction identifiers, i.e. header has something like uint32_t transaction_id, user must choose a unique number (e.g., by increasing it by 1 every time), and the same transaction id is copied to the response. Then there cannot be a context mixup (what query is this reply related to?). Because this is not a lot of work - just copying a number - do this EVEN IF you initially decide to do a very simple synchronous non-buffered request - response protocol which theoretically would work fine without context. User can then choose not to check the id, but then it's their responsibility.

There might be legitimate cases where buffering is needed, though, mostly performance related e.g. batch jobs where you can't accept delays between the steps. But if yours is not like that, just do what nctnico said, it simplifies the business, and the user script can always check some kind of return status code after each step and bail out on failure.

[pcprogrammer]

If I read the OP's intent well, there already is a request-response system, but not synchronous. If a command takes 5 seconds to gather data, I assume it will be send back to the host in some form as a response to the request. This means that the host needs to cope with this data at some point, and the question is if the host can do other useful things in the mean time and if this also implies sending new requests to the device. If not it could just wait for the command to finish.

If the host needs the requests to be processed in order, then it is probably best to do this single request-response cycle at any time and don't cue up commands. If the commands can be processed out of order, a more complex solution could be to evaluate and prioritize the commands, or even handle things simultaneous. Data gathering does not have to fully load the device and this way it might be possible to process quicker commands in the mean time, but like Siwastaja wrote you will then need a request identifier.

Other method would be to send a wait response from the device to signal the host the device is busy processing and a continue response when done. In the days of RS232 and the likes there where the additional hardware handshake lines, but also signals like XON/XOFF within the data handling to get this done.

[Nominal Animal]

  to what nctnico and Siwastaja suggest above.

As an example, consider a text-based protocol similar to G-code or HPGL, where each command is optionally preceded by a numerical ID.  (You can either accept any number within some range that is not already used in a pending command, or require consecutive IDs.)  Commands that are executed immediately return the ID and success or failure, and commands that are handled asynchronously immediately return the ID and operation pending status, and completion status (success or failure) asynchronously afterwards.  Example:
  Request:     1 set pwm 50
  Response:    1 ok: pwm 50
  Request:     get temperature
  Response:    2 pending
  Request:     get frequency
  Response:    3 ok: frequency 2500
  Response:    2 ok: temperature 298

This way, you can also support a cancel request, for example cancel all or cancel 4; a pending request to identify what is still pending; and a wait/wait all or wait 4 command to synchronously wait until all or a specific command has completed before considering any new commands.

The underlying "trick" is to treat the connection as two separate unidirectional connections, with each request and response an atomic/uninterruptible message.  The identifier means they do not need to stay synchronous: while a command is pending, additional commands can be served.  You'll want commands that take enough time for other commands to be processed in the mean time to immediately respond that the command has been started but not completed yet.

Designing the exact protocol to be used is key, to ensure both sides (device and host) are efficient.  Text-based protocols can be "easier" to implement (except for stuff related to newlines and whitespace), but binary protocols are just as reliable using USB Serial and if properly designed can be more efficient (simpler to parse, more compact).  Because of this, I warmly recommend testing it in practice, creating throwaway firmwares with dummy commands and different protocols, until you find one that makes processing on both sides (device and host) robust/easy/efficient.  I personally often play with such protocol ideas in Linux using Unix Domain datagram sockets (but not using the standard library for the communications otherwise).

[SJL-Instruments]

Thanks everyone for the suggestions! Sounds like we have an XY problem - likely we don't need the FIFO at all.

If I read the OP's intent well, there already is a request-response system, but not synchronous. If a command takes 5 seconds to gather data, I assume it will be send back to the host in some form as a response to the request. This means that the host needs to cope with this data at some point, and the question is if the host can do other useful things in the mean time and if this also implies sending new requests to the device. If not it could just wait for the command to finish.

Yes, that's correct. In an older version of the firmware, we did actually implement a blocking request-response model as suggested by nico. Any characters sent after a command was sent, and before the response was complete, would be ignored.

We implemented a FIFO command queue in firmware after a customer requested it. One of the commands returns a sizeable chunk of binary data, so a significant amount of time can be saved by sending the next commands while the previous response is being received.

In hindsight, this doesn't require a FIFO queue - it just requires that new commands can be processed as soon as the previous response starts being sent. (Or a FIFO length of 1.)

As an example, consider a text-based protocol similar to G-code or HPGL, where each command is optionally preceded by a numerical ID.  (You can either accept any number within some range that is not already used in a pending command, or require consecutive IDs.) ...

This system is an interesting idea, although it does add complexity. In our specific case, we don't think it would bring much benefit, as there are very few commands that can actually be executed in parallel. We will keep it in mind for future designs though - thanks!

[nctnico]

A good start would be to look at the SCPI (also known as GPIB or LXI) protocol. Implement that and you'll allow people to write software using standard libraries which take a lot of the low level protocol handling problems away. Ideal for customers who want to interface directly. SCPI also allows to dump data as binary blobs so you don't have to convert everything to text.

Basically, if you design a piece of test equipment which needs to communicate over a byte-streaming interface, implement SCPI. It is designed for that purpose. No need to re-invent the wheel.