I get the technical argument... it's more of a philosophical point, really. For something that's inherently manufacturer-specific like, say, a debug adapter for an MCU, or a piece of instrumentation, then I've no problem with a particular driver being needed.
A COM port, though, is a completely uninteresting, commodity item that either works or it doesn't. The lack of a common interface standard and baseline driver (as a part of the OS) seems like a glaring omission.