Uart Spy? Multi UART tap/debugger?

[paulca]

I'm writing an application that uses several MCUs communicating over UART.

I'm struggling to get debug utilities to work the way I need (at least in Windows, Linux is much easier).   All I would like is a handful, I'd start with 2, UARTRx only probes which interleave them onto a single VCOM/vtty port.  I ideally with auto-baud on both sides.  The other thing I need is a practical non-whining, non-closing, non-annoying terminal applications that will do things like keep a session open, even when the connection closes (PUTTY!), reconnect again when it detects the line go low and doesn't get auto over-ridden by the STM32 Launch output console everytime! (Eclipse!)  /semi-rant.  In linux I can use a command line with microcom or minicom or a dozen other similar applications and at worst have to hit the RESET key combo on a reconnect.  Maybe override a few TTY params.

I have googled and tried about 4 different Windows terminal programs and all but 1 looked like it was written as a final year Uni project in the theory of UART / TTY without actually having ANY practical usage in mind!  Most people recommend Putty, which is quite frustrating when you are dealing with a device which will be up and down more than the price of gas.

I started writing one with 2xRx ports, but something doesn't see right.  Surely this has to have been done before?

A logic analyser.... I hear you.  However mine doesn't work that way.  I have to trigger it, capture the samples for a time period, then it shows me character by character.  It's just too slow.  I need a console window which shows me, reliably what is on those lines.  No "Lost connection...."  non-sense.  Sure print it out, but keep trying!

I thought the AliExpress types would be all over this, but I can find hundreds if different SINGLE Serial adapter boards, but none with even 2.  I figure nobody makes a hardware chip that does that, interleaves (based on char or str delimiter) 2 or more UARTs.

[paulca]

For now...  I'm probably just going to accept defeat and use 2xFTDIs or CP2102s in a breadboard and take up two USB ports.  I suppose I can connect those to the Linux server and access them over SSH more reliably than a windows app.

[8goran8]

Maybe this FT4232 module could help.

https://www.aliexpress.com/item/1005003624740507.html?spm=a2g0o.productlist.0.0.721448d9ePhQDm&algo_pvid=f7f2064b-7acb-4b9d-82e7-d844e57cb787&aem_p4p_detail=202211160239502461041144179280001883710&algo_exp_id=f7f2064b-7acb-4b9d-82e7-d844e57cb787-14&pdp_ext_f=%7B%22sku_id%22%3A%2212000026546798106%22%7D&pdp_npi=2%40dis%21USD%2114.67%2112.03%21%21%213.66%21%21%40210318c216685951901016874e9d18%2112000026546798106%21sea&curPageLogUid=VqfucoHvRRI5&ad_pvid=202211160239502461041144179280001883710_3&ad_pvid=202211160239502461041144179280001883710_3

[paulca]

Indeed, thank you.

Seems my AliExpress search foo was bad.  I had been searching for 2/dual channel UARTs.  I didn't search for 4 channel UARTs!  Nor did I think to search for TTL.

[8goran8]

I have googled and tried about 4 different Windows terminal programs and all but 1 looked like it was written as a final year Uni project in the theory of UART / TTY without actually having ANY practical usage in mind!  Most people recommend Putty, which is quite frustrating when you are dealing with a device which will be up and down more than the price of gas.

Have you tried Tera Term?

[Ian.M]

If you don't actually need four USB COM ports in one chip, there are also modules based on FTDI's FT2232H dual USB UART chip.
Try: https://www.aliexpress.com/wholesale?catId=0&SearchText=CJMCU-2232
The usual concerns apply about the risk of getting fake (functional clone) chips from far east sources, then FTDI crippling them in some future Windows driver release.

N.B. unlike the FT4232 module Goran linked, the CJMCU-2232 does not include level translators.  However although the  FT2232H is a 3.3V device, it has 5V tolerant inputs with a 2.0V Vih threshold.

[paulca]

Ohhh....  interesting. 

As an asides... I just accidentally created a USB death stick LOL.  It seems as though, if you crash the STM32 while it's doing it's very initial USB stuff, and it doesn't get caught by the hard fault handler and instead resets.  It goes into a boot, spam PC, crash loop hitting hte USB host at probably 1000s of Hz for handshakes.  Unconfirmed as the mechanism, but it caused HARD locks in Windows 10.  For a while if I held the reset button on the STM32 micro down I could get Windows to respond.  When I let go it locked up.  Hold NRST and it took it a while to come back.

I was happily torturing it with this and .... it, em... stopped coming back.  In fact anytime I connected any of the USB devices it hard locked.   I had to pull the whole hub just to get to hit Reboot.... which revealed the damage had spread .  No reboot.  Boot fail.  Virtual harddisk error.  Ran "dmesg" and, yep, VirtualBox had even gone as far as crashing the journal on the drive repeatedly until the kernel made the drive readonly.  Absolutely cascade shit storm that required a full reboot of the Host, the guest and I'm still not sure I should plug it back in again. 

I should learn that hard desktop lock ups mean back things are happening at the very lowest levels, don't PLAY with it, reboot it before it destroys a disk.

[ajb]

I don't understand why people recommend putty for anything, it's such a pain in the ass to use.  For UART console, I've been using HTerm and would definitely recommend it: https://www.der-hammer.info/pages/terminal.html

Regarding the weird USB failure, I wouldn't be surprised if the hub was part of the problem.  I had a case where after some sort of electrical glitch in a development unit connected to the PC via a hub I was getting all kinds of random keystrokes and mouse cursor movements, even after unplugging the keyboard and mouse from the hub.  Unplugging the USB connection from the hub to the host stopped it...until I plugged it back in, and the problem continued, even with the actual kb/mouse still disconnected.  Somehow the hub itself got into a weird state where it was just spamming the host with spurious keyboard/mouse activity, even after it should have re-enumerated.  Power cycling the hub finally solved it.  A lot of computers provide some or all of their USB ports via an integrated hub, so it's conceivable that an internal hub IC could experience a similar failure, which, yeah, would end up being a hard reset of the PC kind of situation, whether it's a VM or not.

[paulca]

Generally USB is a ***er.  It just is.  I didnt get to the bottom of what was causing it, but the other issue kept having was random pipeline buffers that would spontaneously start transmitting only after your started to transmit something.  Like even though the VCOM was up, your serial terminal was connected, it would STILL have 2048 bytes sitting in it and as soon as you started to send stuff that would suddenly replay.  The MCU had been powered off and on again and reflashed, so the buffer wasn't there.  Most likely somewhere in the PC USB stack.  I'm sure there is a way to flush/clear and get stuff ready... properly.

It's like learning any peripheral.  You quickly find the tutorial example rapidly grows horns and teeth when you set it lose in the wild.  If you don't implement all those pesky error handlers then it WILL get away from you and you WILL waste hours and hours.

[Scrts]

I thought the AliExpress types would be all over this, but I can find hundreds if different SINGLE Serial adapter boards, but none with even 2.  I figure nobody makes a hardware chip that does that, interleaves (based on char or str delimiter) 2 or more UARTs.

How about 3?
Buy the STM32 blue pill and flash it with Blue  Pill Serial Monster or Pill Serial:
https://github.com/r2axz/bluepill-serial-monster
https://github.com/satoshinm/pill_serial

[westfw]

I have a background project to turn a Seeed Xiao (SAMD21) or Adafruit QT Py SAMD21 into a 3port USB/Serial converter.  Since all (?) Arduino boards expose a UART, an I2C, and an SPI port, and on a SAMD part those are all SERCOM peripherals that CAN run UART on the same pins, the serial part is pretty easy.  I even made a PCB.
I got depressed at the state of the USB libraries, though.  SO MUCH data copying. :-(

https://github.com/r2axz/bluepill-serial-monster
https://github.com/satoshinm/pill_serial

Nice!

[Doctorandus_P]

When I was using windows, I quite liked TeraTerm, but it must have been 10+ years since I last used it.

I also do not quite understand your remark of opening / closing the serial ports. How can an external device close the port? Are you using some sort of handshaking?
I have tried the satoshinm pill_serial project myself for some small tests. I was able to flash it into a blue pill and it created three separate serial ports on my linux box, just by plugging the thing into an USB port. And I could open 3 terminal windows and they could talk to each other, or to themselves by adding some wires on the breadboard. It was only typing some characters though and I have not done any stress testing.

The Teensy 4 apparently has 8 hardware serial ports. I'm guessing that somebody somewhere has written some nice projects for those.

[PCB.Wiz]

For now...  I'm probably just going to accept defeat and use 2xFTDIs or CP2102s in a breadboard and take up two USB ports.

As well as the FT4232 above, there are also
FT2232H  - up to 12Mbd Dual UART HS-USB
CH347     - new Dual UART 9MBd HS-USB part from WCH, modules are available
CP2105   - older dual UART from SiLabs FS-USB
CH9344  - quad UART from WCH but low cost modules not so available ?
CP2108 - SiLabs quad UART, low cost modules not easy to find.

... All I would like is a handful, I'd start with 2, UARTRx only probes which interleave them onto a single VCOM/vtty port.  I ideally with auto-baud on both sides.  The other thing I need is a practical non-whining, non-closing, non-annoying terminal applications that will do things like keep a session open, even when the connection closes (PUTTY!), reconnect again when it detects the line go low ...

A connected sniffer UART should tolerate when the far end pauses HI, but if you unplug the USB to close the session, I cannot see a way to avoid whining 

I don't recall seeing any VCP drivers mentioning autobaud, as most know what baud rate is needed.
The FS-USB ones mostly auto trim a RC clock to the usb frame and so define baud to ~ 0.25%, and the HS-USB ones all use crystals  so they are tens of ppm error

Only poor clock tolerance links like LIN Bus really need Autobaud

[DavidAlfa]

Can't you use a stm32 virtual port?
Rs232 is rarely used in full duplex mode, so you be able to easily  format the output like:

A: Hello there
B: Hi Chris
A: Gimme the hex bits please
B: Live is so hard... Where will we be in 5 years?
A: Stoned again? Reset, you b*stard!
B: Reset done
A: Gimme the hex bits
B: Escusez-moi, je n'ai entendu
A: ?? ??

[westfw]

How can an external device close the port?

If it's a USB device emulating a COM port over CDC/ACM, the port "disappears" when the the device resets or something goes wrong with the USB.  Annoying.

There's a whole class of products in the "console server" arena, designed for monitor output from multiple serial ports (or TCP connections), designed for monitoring a bunch of (you guessed it) UART console ports of servers, routers, switches, and so on.

See, for example:
https://www.conserver.com/
https://www.conserver.com/consoles/

[edtyler]

At the far end of the cost spectrum Lecroy makes a very good serial sniffer that uses USB to connect to the PC. The software records each transmission and a timestamp. Both sides' transmitter data can be displayed in an interelaved fashion so that race conditions in the protocol can be easily seen and debugged.

Great tool. Price > $1,000 USD. However, it has saved my customers many thousand of $ by quickly and clearly showing what is happing on a serial link.

[paulca]

Can't you use a stm32 virtual port?
Rs232 is rarely used in full duplex mode, so you be able to easily  format the output like:

A: Hello there
B: Hi Chris
A: Gimme the hex bits please
B: Live is so hard... Where will we be in 5 years?
A: Stoned again? Reset, you b*stard!
B: Reset done
A: Gimme the hex bits
B: Escusez-moi, je n'ai entendu
A: ?? ??

That was the plan with the STM32Black pill.  I almost had it working, but I was trying something different, "chasing the dma write pointer"  because I can't wait for a "half cplt callback" if the UART just sent "OK\r\n"

So I fire a timer every millisecond, take a copy of the dma remaining counter register and go chasing up to it looking for new lines.  Copying the messages they delimit into a transmit buffer when I send them.

Works.  For a while.  Then it tramps off around the DMA buffer again outputing all the messages it finds in it again.  No idea how, no idea why.  Needs rewritten it's got issues.  I mean the full negociation of my device is over in 1.5 seconds, but for 10 seconds after that my device is STILL finding OLD messages in the buffer and replaying them in little bursts.  Perfectly formed too.  Some how it's overtaking the write pointer and extracting all the old messages from the dma buffer.  It's so odd the way it replays them in little bursts too.  So, yes, issues.  Its my own fault, I'm at the STM32 stage where I keep trying new things and am maybe a little less diligent about my buffer protection than I should be.  I keep trying to make it as simple as possible and ending up failing.  So maybe I should just go straight to what I know works and asynchronous Get/Release buffer handling.  Stops issues updating the head/tail pointers which is where I think I'm failing in the current UART Spy Pill.

[paulca]

How can an external device close the port?

If it's a USB device emulating a COM port over CDC/ACM, the port "disappears" when the the device resets or something goes wrong with the USB.  Annoying.

There's a whole class of products in the "console server" arena, designed for monitor output from multiple serial ports (or TCP connections), designed for monitoring a bunch of (you guessed it) UART console ports of servers, routers, switches, and so on.

See, for example:
https://www.conserver.com/
https://www.conserver.com/consoles/

The Eclipse serial terminals are actually the closest I have found to what I want.  Actually the issue with them is they are like a dog with a bone.  If you create a serial terminal for COM10, then it will continually try to connect to anything on COM10.  When you have been working for a few hours and the device has changed COM port a few times, you end up with 3 or 4 serial terminals, all trying to connect to various COMN ports.  Even if you don't have those serial "consoles" visible or even active.

This is what made the USB lockup crash so much harder to get of out as it was ALL of the those eclipse consoles pinging all the VCOM ports as soon as they appeared.

The STM32 current VCP COM driver supports autobaud.  You can set it in the USB desc, but it's left blank for auto.  Basically you can connect the terminal to the COM port at any speed you like and it works.  I did try 20Mbit/s or something stupid which eclipse offered to me and the VCOM port said "Nope".  But 912xxxx it's fine with.

Another interesting feature I think I have found on the STM32 UART drivers, it doesn't seem to allow, or at least filters out, out-of-baud transmissions.  Normally if you connect a 9600 baud signal to a 115200 baud port you get random garbage.  When I did this to my current setup, it just muted it and ignored it until I set it to the right speed.  I don't know if that's intentional or I just got lucky.

[abyrvalg]

UARTs are pretty slow, compared to STM and USB speeds, so you could just enable per-byte RX interrupts (or even poll RX complete flags sequentially in a “main loop”) and move bytes to a common ping-pong USB buffer together with port number (and a time stamp taken from some timer, if you want it). This way you’ll get everything nicely related if that’s important.

For simpler cases with slower devices sending command packets and getting response packets after some time in an order I’m just hooking two CP2102-based adapters (they seem to have lower latencies than FTDIs) and receiving packets with a short Python script consisting of two async packet parsers in an endless loop. I have this setup right now on a table, dumping 115200 link between a dashboard and an ESC of a e-scooter.

The VCP “autobaud” you’ve mentioned has nothing with real UART baudrate detection. The USB side of a VCP link always runs at full speed, the baudrate is just sent to the device in a control packet and it’s up to the device what to do with it - reconfigure the UART side or just ignore (you have the latter I guess).

[paulca]

I know.  When I started the thing I kept telling myself to just clock in characters in the interrupt.  But, I had to try something else.

Ironically  that's more or less exactly what the application I am testing does!

[DavidAlfa]

STM32 VCP doesn't care about baudrate.
I can open the terminal, set either 1 or 98346732 bauds, will work in the exact same way.

[abyrvalg]

The mentioned Python script, just in case

[SiliconWizard]

STM32 VCP doesn't care about baudrate.
I can open the terminal, set either 1 or 98346732 bauds, will work in the exact same way.

Like any VCP implementation that doesn't actually use a physical UART interface. Then the baudrate has absolutely no meaning and the USB communication will happen at whatever speed it can handle between host and device.

The baud rate is only used to configure a physical UART if the USB device implementing a serial port class controls one.

[DavidAlfa]

I know, I was replying to paulca saying the port didn't work at weird baudrates.