COM-Ports are very annoying - cant connect to comport / not available

[eTobey]

When i program my microcontrollers, i use a lot of serial monitor and own programs to monitor and control them. But very often, i get problems, that i cant connect to them. Sometimes it helpes to close the programs that used them, reconnect the and reconnect the usb, but sometimes it doesnt. I have 4 programs that connect to either one of the 2 COM ports.

Any help ideas would be much appreciated. 

[DavidAlfa]

Normal behavior, you can't access a com port used by another program.
So, you must disconnect/close that program before other can use it.
Nothing can be done about it.

[Ian.M]

Worse than that: On Windows, using USB devices that enumerate as a COM port, you can run out of free COM ports, and Windows can fail to attach any COM port to the device!

Many MCU development boards and programmer/debuggers that enumerate as a COM port (or as a composite device including a COM port) don't have a unique serial number in their USB device descriptor, which results in Windows tracking them by where they are on the USB device tree.  Move them to a different host or hub port or add or remove a hub between them and the host, and Windows creates a new persistent instance of the device.  Instances with no device physically connected get hidden, but can still 'squat' on a port number.

See https://theitbros.com/how-to-delete-com-ports-in-use/ for how to diagnose and clean up the resulting mess.

Mitigation: Try to always connect your development tools to exactly the same USB port.

[DavidAlfa]

Yes, when reaching com256 it will break.
Happens a lot at work when testing boards with USB serial chips.
Not only that, the testing program would store the last com and instantly crash, requiring the settings to we wiped.
I had to make a script for that using COM Name Arbiter Tool.

[spostma]

Probably you know already, but to open COM port number > COM9 you need an escape sequence like '\\.\COM99', see:
https://support.microsoft.com/en-us/topic/howto-specify-serial-ports-larger-than-com9-db9078a5-b7b6-bf00-240f-f749ebfd913e

If you run out of COM port numbers because of device testing during production,
then you can disable USB serial numbers either globally or per USB VID:PID combination, see:
https://ftdichip.com/Support/Knowledgebase/index.html?ignorehardwareserialnumber.htm
https://forums.tomshardware.com/threads/keep-having-to-resintall-printer-driver.1010374/

Of course, you also can simply backup the registry and restore it after all testing is finished.

I use a very old and very handy tool called 'Device Remover' from 02.09.2009 written in .Net 2.0
to selectively clean up registry entries for removed devices (it was on a CD of the German C't computer magazine)
It seems you still can download it from https://device-remover.updatestar.com/

In the Windows Device Manager, you also can select 'show hidden devices',
and then uninstall any COM port that you do not use any more.

Finally, there is an excellent tiny tool called Com Port Manager written by Uwe Sieber; do check it out:
https://www.uwe-sieber.de/ComPortMan_e.html

Hope this helps you...

[Halcyon]

I can't say I've used an old-fashioned serial port on a modern computer, however I remember back in the day, serial ports would share common IRQs. e.g.: COM1 and COM3 shared then same interrupt, so only one or the other could be used at the same time (well they kind of could, but behaviour was erratic and unpredictable).

This might sound silly that I don't know the answer, but are these kinds of IRQ conflicts a thing anymore? It's not something I've turned my mind to in the last 20+ years.

[langwadt]

I can't say I've used an old-fashioned serial port on a modern computer, however I remember back in the day, serial ports would share common IRQs. e.g.: COM1 and COM3 shared then same interrupt, so only one or the other could be used at the same time (well they kind of could, but behaviour was erratic and unpredictable).

This might sound silly that I don't know the answer, but are these kinds of IRQ conflicts a thing anymore? It's not something I've turned my mind to in the last 20+ years.

unless you break into a technical museum and steal computer it's not an issue, most comports these days are USB

[Halcyon]

I can't say I've used an old-fashioned serial port on a modern computer, however I remember back in the day, serial ports would share common IRQs. e.g.: COM1 and COM3 shared then same interrupt, so only one or the other could be used at the same time (well they kind of could, but behaviour was erratic and unpredictable).

This might sound silly that I don't know the answer, but are these kinds of IRQ conflicts a thing anymore? It's not something I've turned my mind to in the last 20+ years.

unless you break into a technical museum and steal computer it's not an issue, most comports these days are USB

I realise that this isn't an issue on USB, but I missed the bit where the OP mentioned this. For whatever reason I had it my head that he was using some kind of serial IO card or interface, other than USB. My bad. I should read properly.

[golden_labels]

This might sound silly that I don't know the answer, but are these kinds of IRQ conflicts a thing anymore? It's not something I've turned my mind to in the last 20+ years.

Not anymore. PCI Express uses message signaled interrupts instead of physical IRQ lines. PCI could use either MSI or physical lines, but — unlike with old ISA — they were designed to be shared and numbers were assigned in software. And nowadays serial ports are exposed either over PCI Express or even as virtual USB serial devices. So the limitation is in how the operating system handles them.

The problems usually arise, because either the port is used by another program, because it retains configuration from a previous program, because the connected device misbehaved and has to be reset, or because the port got renamed. I don’t know about Windows, but at least in Linux systems with udev it’s possible to set rules to give persitent names to specific devices. This solves the last problem.

[Halcyon]

Makes perfect sense. Thanks for the explanation. As I said, it's not really something I've given thought to recently.

[Whales]

From the other side of the ditch (Linux) there are some annoying things too.

For context: on Linux your serial ports are a file (like "/dev/ttyACM0") that you configure and then read & write to.  On Windows they're a file that has no path like "COM1", this is why you can't save a file called "com.txt" (and a whole bunch of other names) anywhere on Windows.  Try it now if you don't believe me 

(1) Naming schema is more complex than windows

/dev/ttyACM* <- some USB uart dongles
/dev/ttyUSB* <- some other USB uart dongles (different driver?)
/dev/ttyS* <- RS232 built onto your motherboard

I know all these names off by heart now, but if you don't then it's a bit annoying to go looking for them.  Especially since most systems have 64 dummy /dev/tty* entries that you need to ignore.

On the other hand: it's sometimes kind of useful to have different brands of USB uart under different names.  I board I was debugging recently had two serial interfaces, one of which would flap (see the next problem), so this made reconnecting a bit more predictable.

(2) Dealing with serial ports that appear and disappear. 

Let's say you have /dev/ttyUSB0 open in your serial monitor, but the board you are talking to implements this serial port in software over its usb interface.  When you reset the board (eg flash it) then your desktop machine's kernel will notice the USB device is gone and will tear down /dev/ttyUSB0.

Now about a second later your board boots up again and re-initialises the software UART over USB.  Your desktop kernel notices and creates /dev/ttyUSB1, because the zombie remains of /dev/ttyUSB0 might still open in some other program and it doesn't want to upset it.

:|

I usually have to do something like this:

Code: [Select]

while true; do picocom -b 115200 /dev/ttyUSB*; sleep 0.2; done
(3) Software that auto-opens serial ports.

I run a 3D printer off my desktop.  It uses a USB serial interface.  Guess what happens if some random piece of software starts trying to write to that serial interface halfway through a multi-hour print job.

"Cura" is particularly bad.  I accidentally opened it one day (for some reason it had become the default file opener for .stl).  My existing print job got trashed.    I think Cura also decided to dance with my board debugging session one day.  I've uninstalled it now so I can't make this mistake again



I will say however that overall things work a bit smoother with serial ports on Linux than Windows.  I don't have to close and reopen ports all of the time (and I don't know why this is even a thing that needs to happen on Windows or what causes it).  I'm really curious if anyone knows anything more.

[Whales]

Hmm, I wonder if some other program on windows is probing/opening the ports?  Could perhaps Systinternals Process Monitor help track down the culprits?

[golden_labels]

As I said, it's not really something I've given thought to recently.

I hope I didn’t make you feel bad for not knowing this! That wasn’t my intention. Just sharing.

I am myself not an expert. The only reason, I know this, is because I re-read Norton’s “pink shirt” book⁽¹⁾ recently and, to satisfy my curiosity, I checked how the things are handled now. And this is really superficial knowledge on my end.

I usually have to do something like this:

Code: [Select]

while true; do picocom -b 115200 /dev/ttyUSB*; sleep 0.2; done

udev to the rescue! An example rule for USB device vid:pid 0403:6001, with serial number “A7035UBT”:

Code: [Select]

SUBSYSTEMS=="usb", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="A7035UBT", GROUP="users", MODE="0666", SYMLINK+="some-persistent-name"Does not solve all trouble, but at least you get a persistent name. Here also widening access rights.

⁽¹⁾ Or a very weird artifact of unknown origin and history, which I found in boxes with books: an almost certainly unauthorized Polish translation of the book. Weird, because this is a print made on good paper, properly bound, with somebody meticulously correcting mistakes with a ballpen. Even includes a coffee stain.

[magic]

I can't say I've used an old-fashioned serial port on a modern computer, however I remember back in the day, serial ports would share common IRQs. e.g.: COM1 and COM3 shared then same interrupt, so only one or the other could be used at the same time (well they kind of could, but behaviour was erratic and unpredictable).

I'm not familiar with the dramas of early PCs, but the usual way of dealing with shared interrupts is polling every device connected to the line whenever the interrupt is raised in order to find out which one(s) need attention. So I'm not sure what the problem even was.

The Linux serial FAQ says:

When two devices have the same IRQ number it's called sharing interrupts. Under some conditions this sharing works out OK. Starting with kernel version 2.2, ISA serial ports may, if the hardware is designed for this, share interrupts with other serial ports. Devices on the PCI bus may share the same IRQ interrupt with other devices on the PCI bus (provided the software supports this).

Linux 2.2 is previous century stuff, so hopefully it's not an issue today. But I think I only have one computer with (maybe) two serial ports and never tried using two at a time.

PCI and PCIe are irrelevant since they only serve as glue between the CPU and the ISA SuperIO chip which contains the UARTs. PCI can deliver ISA interrupts to the CPU and surely enough, PCIe can tunnel legacy PCI interrupts to the CPU too. PCIe devices can even originate emulated legacy interrupts themselves in order to support software that doesn't know MSIs.

Maybe something has changed in the last few years with the introduction of x86 SoCs, but I'm not sure. They could as well pull the whole southbridge and SuperIO logic unmodified onto the CPU die and call it a day.

[nali]

Yes, when reaching com256 it will break.
Happens a lot at work when testing boards with USB serial chips.
Not only that, the testing program would store the last com and instantly crash, requiring the settings to we wiped.
I had to make a script for that using COM Name Arbiter Tool.

In Windows there is a registry entry to stop multiple devices with the same VID:PID but differing serial numbers registering new COM ports which is handy for test setups, see this FTDI article https://www.ftdichip.com/Support/Knowledgebase/index.html?ignorehardwareserialnumber.htm

[Nominal Animal]

(1) Naming schema is more complex than windows

I use udev rules to create symlinks to my microcontrollers, so they have stable names.
It only takes a single line in a plain text file, saved (with superuser privileges) to the correct location.

(2) Dealing with serial ports that appear and disappear.

In the case of a udev symlink, you still need to reopen the same device (symlink).  This cannot really be avoided.  Attempting any I/O to the disappeared device will result in a clear error, though.

(3) Software that auto-opens serial ports.

There are two details here.

One is to use an udev rule that tells services like ModemManager to skip the device.
The other is to use the standard TTY ioctl, TIOCEXCL, in the program using the serial port.  This stops all but superuser from opening the port while the program has it open; they fail with EBUSY (Device or resource busy) error.

(It is just a single call, ioctl(fd,TIOCEXCL); or ioctl(fileno(stream),TIOCEXCL);, after the port has been opened.  This will stop all except system services like ModemManager that do run with superuser privileges; those are controlled via the udev rule.)

"Cura" is particularly bad.  I accidentally opened it one day (for some reason it had become the default file opener for .stl).  My existing print job got trashed.    I think Cura also decided to dance with my board debugging session one day.  I've uninstalled it now so I can't make this mistake again

You need to use a program that does the TIOCEXCL ioctl to communicate with your boards or printers.

The main downside of the way Linux routes serial and USB serial ports through the tty (termios) layer is that it adds overhead, limiting the throughput somewhat.  For example, on Teensy 4.0 and 4.1 (high-speed USB, 480 Mbit/s), reading USB serial data from Teensy is limited to 25-30 megabytes per second (200-230 Mbits/s), depending on the single-core CPU speed and USB hardware implementation.


I use C and Python to interface to my microcontroller projects, using POSIX termios (tty) interfaces (man 3 termios for C, or python termios built-in library).  Python isn't suitable for massive amounts of I/O, say more than a couple of megabytes per second, because of the inherent complexity in its I/O implementation, but it is definitely sufficient for full-speed USB serial (12 Mbit/s).

By default, the termios layer does all sorts of transformations ("canonical mode") and can even generate signals (INT, QUIT, TSTP, TTOU; and INFO on BSDs but not in Linux), and these need to be turned off ("raw mode").  It's just a dozen lines of code or so, fortunately.  The annoying bit is that to be "nice", one needs to store the original settings, and restore them before exiting; while not required per se at all, it is kinda-sorta expected.

One thing I'm adamant on in my own microcontroller implementations –– I use several different microcontrollers, all with native USB implementations, from 8-bit AVR (ATmega32u4) to i.MX RT1062 –– is that the communications are truly asynchronous, even for a query-response interface.  That means that the connection is essentially two separate unidirectional "pipes" that work independently, not in lock-step.  One can supply more than one query in a row, without waiting for the response in between.  This is also why e.g. G-code supports line numbers (Nlinenum).  In my own query-response interfaces, I similarly allow an unique identifier for each query or command, repeated in the response; and explicitly allow (in my host program) responses to occur in a different order.  This way, operations that take longer, say moving a servo, do not delay faster operations, say changing the state of a relay.

A particularly useful approach I've found is to use Qt5 or Gtk+ (via PySide2 or PyQt5, or Python GI) for a nice graphical UI, with a separate thread (or thread pair!) communicating with each microcontroller, python Queues between the threads passing the commands and responses and data around, and an idle handler converting the messages in the response Queues to UI events.
This way, if you have say a button for a relay, clicking on the button does not immediately change its state, only sends the Queue message instructing the MCU to change the relay state.  When the MCU has done so and responds, the response message is what changes the button state.  (I personally like to use my own "button" derivative class, though, with an additional mid-way state (immediately when pushed), showing the user the button is in the process of changing,  but not done changing yet.  You could use a push-button and a separate indicator, for example, with the button popping back up when the response is received and the indicator updated.)

To make such an application portable, the low-level serial interface (basically dual pipes to the microcontroller) has to be abstracted out.  I really dislike libusb (I don't trust it, it just ignores most errors internally) and other serial abstractions I've seen.  The termios approach works for all but Windows, although how one discovers the suitable microcontrollers/serial devices does vary a bit between Linux/Android and BSDs.  Annoying, but such is life.

[Nominal Animal]

The Linux serial FAQ says:

When two devices have the same IRQ number it's called sharing interrupts. Under some conditions this sharing works out OK. Starting with kernel version 2.2, ISA serial ports may, if the hardware is designed for this, share interrupts with other serial ports. Devices on the PCI bus may share the same IRQ interrupt with other devices on the PCI bus (provided the software supports this).

Linux 2.2 is previous century stuff, so hopefully it's not an issue today. But I think I only have one computer with (maybe) two serial ports and never tried using two at a time.

It depends a bit on the exact serial port hardware, but generally speaking, if the hardware supports two or more ISA or PCI serial ports, the driver supports IRQ sharing.  Multiple hardware serial ports on industrial/automotive boards using AMD/Intel CPUs, using 8250/16550 serial drivers, were very common in industrial/automotive embedded devices running Linux back in 2000-2010; haven't kept up with the state of things since.  In the 1995-2000 era, the PC motherboards I used, tended to have two 16550-based serial ports, and I seem to recall they worked just fine concurrently in Linux.

IIRC, Windows 95/98/2000/NT used a single driver on those typical two-16550-based-serial-ports motherboards for both ports, so that the IRQ sharing was not an issue.

The one problematic case I know of, is when you have two different chips for the two main ISA serial ports – say, one on the motherboard, and one on an ISA expansion card.  The Linux legacy 8250/16550 driver core can share IRQs between drivers, but not all serial drivers can/could.  Basically, the IRQ had to be chained, and the driver be ready to do nothing (just forward the IRQ down the chain) when interrupted.

[PlainName]

Normal behavior, you can't access a com port used by another program.
So, you must disconnect/close that program before other can use it.
Nothing can be done about it.

That can be worked around by having a virtual serial port that connects to the real serial port and then shares with whatever applications talk to it. There are a few of these and picking on at random is:

https://freeserialportsplitter.com/

Edit: the one I was trying to think of is com0com on Sourceforge. Does many thing like looping the output of one virtual port to the input of another, so one application can provide the input for another (great if you're developing a tool to talk to embedded kit via serial port, for instance). Multi-application access to a single port is just one thing it does.

[Ian.M]

+1 for com0com.  I use it for serial redirection over TCP/IP on Windows so I can monitor and reprogram an Arduino based robot remotely. https://www.eevblog.com/forum/beginners/connecting-arduino-to-computer-with-wifi-instead-of-usb/msg3754835/#msg3754835

[golden_labels]

I'm not familiar with the dramas of early PCs, but the usual way of dealing with shared interrupts is polling every device connected to the line whenever the interrupt is raised in order to find out which one(s) need attention. So I'm not sure what the problem even was.

In ISA each IRQ was a physical line. These were going directly to a 8259 controller, which was then informing the CPU. Later a second 8259 was added, acting as a device signaling to the first one on IRQ2. While the 8259 supports both level- and edge-triggering, ISA only used the latter. That made it impossible to detect and reliably handle more than a single interrupt on the same line. Even if level-triggering was used, the 8259 only knew about the interrupt lines, so it alone would still not be able to tell the source. This is in contrast with later, more complex PC host controllers, which together with operating system (and drivers) collectively have some oversight of the hardware. The problem was exaggerated by almost all interrupt lines having functions assigned by default. And on top of that all expansion cards having IRQ lines either hardwired or at best jumper-wired, with nothing in the system knowing, what they use.

In PCI there are only 4 IRQ lines. But the entire system (hardware and software) is designed, so it can distinguish between devices. And in fact is required to do so, which was not a requirement with ISA (even if there existed hypothetical solution). PCI does not have direct mapping between IRQ lines and IRQs, and the IRQ numbers may be assigned in software. So, until a device/driver is badly designed, potential conflicts are resolved at the OS level.

[magic]

While the 8259 supports both level- and edge-triggering, ISA only used the latter. That made it impossible to detect and reliably handle more than a single interrupt on the same line.

Okay, I can see how some contrived situations may cause problems here: if two devices raise an interrupt, and the first to be serviced raises another interrupt before the line is released by the second device, the second interrupt of the first device may be lost unless the kernel runs all registered handlers alternately in an infinite loop

I have also read somewhere that the "hardware support for ISA IRQ sharing" mentioned in the serial FAQ supposedly means pulsing the IRQ line for a short time to avoid creating the aforementioned condition. The software side of course involves running all registered handlers when a shared interrupt is received.

Unfortunately I don't know how common IRQ sharing support is on modern motherboards, but since they don't have ISA slots you shouldn't run into issues. I think native PCI UARTs will use native PCI interrupts for their ports so these ought to work in any number. And of course no problems with USB and so on.

... unless your OS only supports 256 (or is it 255?) of them

[garrettm]

So, just a thought. I use an Ethernet to serial server (MOXA 5650-8) to access my serial devices. This method allows for multiple machines to connect to the same serial device either on the LAN or through the internet and greatly simplifies connecting old HW to new HW (say an ultrabook with only USB-C ports) and is OS independent (if you use a TCP socket--emulating a COM port does require a driver though). The only real down side to this method is that it can be slower than a direct serial connection due to the nature of Ethernet signaling (packets) and TCP overhead and possibly changing code to work directly through a TCP socket rather than a COM port (assuming you choose not to emulate a COM port).