Minimal USB CDC for PIC32...Update: Say goodbye to COM ports

[voltsandjolts]

In the past I have always used one of the dedicated USB-Serial converter ICs (FTDI/Prolific/CH340 etc) for serial comms.
I have a fairly simple project using PIC32MX795 which would be suited to using the USB peripheral instead of a converter IC.
So, I fired up Harmony for the first time (normally I would just use the datasheet for PIC32 programming) and tried out the USB option.
Harmony automatically pulls in FreeRTOS as a dependency

Before I trawl through the Harmony code to sift out the crud, can anyone point me to a minimal PIC32 USB CDC implementation?

For example, someone has done a nice job of this for PIC18, one source file and a few headers, et voila. Nice.
https://pic-microcontroller.com/a-minimal-usb-cdc-acm-aka-virtual-serial-port/

[JPortici]

Yes, check Mister Hemi's work https://www.microchip.com/forums/m1083508.aspx

[voltsandjolts]

Thank you, that looks very promising, one source file and one header!
I will give that a try.

[Edit: I looked at it but it's just so wrong its silly. Ignore Mister Hemi's code.]

I see you responded a few times in that thread - did you get it running?

I wish manufacturers would provide a simple CDC implementation for their USB peripherals, surely that would cover a lot of use cases where folks are looking for an easy UART equivalent over USB. It really shouldn't be this time consuming.

[JPortici]

I actually did not, because i always wanted to use USB on different targets (MX and MK) which use different controllers and i didn't know how to port it.
For the MX i went on using the good old MLA, as i'm doing in other products using dsPIC/32MM.. performance is more than acceptable.
For MK, project was put on hold indefinetly (hobby stuff)

[voltsandjolts]

Ahh, I didn't realise the USB controllers were different, I had just assumed they'd be the same across PIC32
I'll probably just switch micro from MX to MZ, easier than porting USB stuff from MZ to MX.

[voltsandjolts]

Can someone give me some guidance on which are the newer/improved PIC32 MZ micros?
They had a bad reputation for being buggy in the early days.

I am looking at a MZ EF series, 512K/128K in 64 or 100 QFP...or such like.

[JPortici]

PIC32MZ EC: The old bugged ones
PIC32MZ EF: Those which replaced the EC, with also a FPU
PIC32MZ DA: PIC32MZ EF, but with also a graphics and SDRAM controller. selected versions have also up to 32MB of SDRAM in the same package

PIC32MK: Promising on paper, but implementation sucked. Not nearly as bugged as the MZ EC but has some dealbreaker bugs (VBAT not working, Prefetch not working and can lock up processor, DMA error can lock up processor, single SRS for interrupts) supposedly solved in the new versions (which will replace the old ones, which will add ECC flash, CAN FD and more improved stuff)

PIC32MX: Several families, with all their quirks. However i quidte like the 1xx/3xx for cheap CAN+USB

PIC32MM: Has some dealbreaker bugs (namely ADC with missing codes) but overall quite pleasant for reeeeally cheap usb

[voltsandjolts]

Great summary, thank you 

That should be on front page of Microchip website 

[Renate]

If you like, you just do the FTDI implementation of a serial interface.
It's simpler that the spec USB CDC.

[JPortici]

or better yet, don't use CDC at all.
Low bandwidth, i just use one or more interrupt endpoints (custom HID device) so no driver needed.. and 64kb/s transfer speed is usually plenty, even compared to UART
High bandwidth, or packets bigger than 64 bytes, i prefer to go the vendor route (one or more bulk endpoints and be done with it)
Sure, going CDC means that you don't have to deal with libusb/winusb but then you have to deal with all the limitations of COM ports under windows. Don't know if there are simillar nuisances with linux/mac, don't care

[Renate]

(custom HID device) so no driver needed..

Oh, yeah, definitely!
I mostly do HID, the only "serial" I use is debug/console.

Everybody is always using serial. It's unstructured. It stinks.
Collect characters until newline or timeout or did you miss the start of the sentence?

[dmendesf]

If you like, you just do the FTDI implementation of a serial interface.
It's simpler that the spec USB CDC.

You mean using FTDI drivers with a firmware emulating a FTDI device inside a microcontroller? Do you know how they transfer data?

[voltsandjolts]

or better yet, don't use CDC at all.

Yeh, COM ports are shit aren't they.
USB CDC was created for legacy equipment which used COM ports.
At that time a more modern standardised replacement for COM ports should also have been implemented, with a simple API maybe like this:

ListDevices( 256-bit unique device type ID )
OpenDevice()
CloseDevice()
SendPacket( bytes )
GetPacketByTimeout( timeout )
Callback for received packet ( *fptr )
Callback for device disconnected ( *fptr )

So, cross-platform drivers available on every system, no baud rate crap, no searching through COM ports for your device, simple packetised data.
Our technology leaders failed us - looking at you USB-IF.
So here we are in the year 2020, basterdising the USB HID class and many of us still using frickin COM ports.

It actually makes me mad that, for the want of something so simple, literally thousands of engineer hours (and user hours messing with driver installs) have been wasted on simple serial comms over USB 

FTDI with the DLL interface works good for me but something similar should have been made available as a standard USB class with standard driver many years ago.

[JPortici]

USB CDC was created for legacy equipment which used COM ports.

I would not be surprised if keeping visual basic alive was also a reason.
After all, i am still mantaining a 20+ yo application that went through VB4,5 and 6 which i can't really migrate to a new framework because the budget to do that is zero. USB CDC and classic bluetooth means that i can still use a COM port and not have to come up with monstrous ideas like having another process running written in a more up to date framework such as QT and communicate between the two processes via winsocket or something like that so i can finally use real USB and BLE (though BLE support in windows is still atrocious and i prefer to have an android tablet and a specific app for such occasions)
Then we have spare laptops that run XP or worse 98 because there is some piece of software written in the 90s that was never updated with 32bit binaries, that will not work inside a VM because reasons (sometimes windows XP mode in win7 worked though)

[Renate]

You mean using FTDI drivers with a firmware emulating a FTDI device inside a microcontroller? Do you know how they transfer data?

I must know. Or did know. I wrote a replacement once when I didn't have an actual FTDI around.

The Windows driver (written by FTDI) is more particular as FTDI tries to prevent your using clones.
The Linux driver is easy-peasy.
All the info you need can be found in the header file for the Linux driver:
https://github.com/torvalds/linux/blob/master/drivers/usb/serial/ftdi_sio.h

It's two bulk endpoints and the control endpoint is used for vendor specific control/status.
The OUT transactions are just raw, the IN transactions have two bytes at the head, 0x01, 0x01.

The good thing is that you don't need to really implement the parts you don't need, baud rate, 5-8 bit data size, parity, etc.
You also only need to implement a single model.

[dmendesf]

Thanks for the info. Will try someday...

[voltsandjolts]

...and 64kb/s transfer speed is usually plenty, even compared to UART.
High bandwidth, or packets bigger than 64 bytes, i prefer to go the vendor route...

Everybody is always using serial. It's unstructured. It stinks.
Collect characters until newline or timeout or did you miss the start of the sentence?

My preferred solution for packetisation over serial links is the simple and elegant COBS encoding.
https://www.eevblog.com/forum/microcontrollers/implementing-uart-data-packets-with-consistent-overhead-byte-stuffing-(cobs)/

So if we can use COBS to send arbitrary length data packets over USB HID (broken up into 64 byte chunks), that leaves the speed limitation of 64,000 bytes/sec (in each direction simultaneously, I believe..?)

In comparison to good-old-fashioned COM ports, that would need a 8N1 baud rate of around 640kbaud for same data rate.

Hmmm, maybe that would suffice for my next project. I need to look into this HID stuff a bit more closely.

[voltsandjolts]

Ahha, found a neat library:

M-Stack: Free USB Stack for PIC 16F, 18F, 24F, and 32MX Microcontrollers
https://github.com/signal11/m-stack
LGPL or Apache license.

Includes examples for CDC, HID, MSC.
Just a few minutes after downloading I had the CDC example running on a PIC32MX795F512, using just under 12K FLASH and 1K RAM (-O2 free XC32 version)

All self contained.
No hunting for old Microchip libraries or compilers, it just compiled first try on MPLABX 5.45 and XC32 v2.5 (latest versions as of November 2020)
That's refreshing!

[JPortici]

It does? I never was able to use that one

[voltsandjolts]

Okay, I've been looking into this a bit more and I've come to the conclusion that I don't need CDC and COM ports anymore, woo-hoo!

Firstly, I looked at using USB HID which is a nice solution because there is no driver install.
usbhid.sys/usbhid.dll are used as kernel/userspace drivers and hidapi.dll[1] provides an easy to use cross-platform API.
The maximum data rate of 64,000 bytes/sec full-duplex is ample for many applications.

..but what I've recently discovered (many of you probably knew years ago) is that you can use any USB transport type without a driver install and get FS/HS/SS data rates. Just as easy as HID.
With Windows 8.1 onwards, your USB device can respond with certain string descriptors which cause it to be assigned winusb.sys/winusb.dll as the kernel/userspace drivers, with no user interaction.
Plug and play. Your host software can then use winusb.dll directly, or via libusb for easier API and cross-platform ability.
This system is often referred to as Windows Compatability ID (WCID). There are some links below [2] to get you started if you want more background info.

There is an open source embedded USB stack called 'M-Stack' [3].
It's for all PICs which have the Microchip Full Speed USB peripheral, and that includes PIC16/18/24/32, not PIC32MZ (which has HS USB).
IMHO it's well written and much easier to understand than any of Microchip's USB code, which has so much abstraction and pointer shenanigans it makes me dizzy.
Useful to look over even if you are going to use Microchips code in your device.
Anyway, M-Stack includes this WCID feature and I'm attaching a ready configured project here.

FYI M-Stack needed one tiny fix, the last byte of this struct was zero but it actually must be 0x01:

Code: [Select]

struct microsoft_os_descriptor os_descriptor =
{
  0x12,                          /* bLength */
  0x3,                           /* bDescriptorType */
  {'M','S','F','T','1','0','0'}, /* qwSignature */
  MICROSOFT_OS_DESC_VENDOR_CODE, /* bMS_VendorCode */
  0x01,                          /* bPad             MUST BE 0x01  */
};

This project has been tested on PIC32MX but should work for other PIC families.
Change MPLABX project settings and hardware.c for your device as appropriate.
Build the project (no massive external dependencies like Microchip USB stuff, all self contained) and program your device.
Plug it into a USB socket and it appears in Win10 Device Manager under 'Universal Serial Bus Devices' as 'USB Stack Test Device'
It is setup with VID=0x1209 PID=0x000E which is assigned for test devices by pidcodes [4].

USB Tree View[5] shows all the details for the device, with the four endpoints established (pipes), in addition to endpoint 0.

Code: [Select]

---------------- Connection Information ---------------
Connection Index         : 0x06 (Port 6)
Connection Status        : 0x01 (DeviceConnected)
Current Config Value     : 0x01 (Configuration 1)
Device Address           : 0x0E (14)
Is Hub                   : 0x00 (no)
Device Bus Speed         : 0x01 (Full-Speed)
Number Of Open Pipes     : 0x04 (4 pipes to data endpoints)
Pipe[0]                  : EndpointID=1  Direction=IN   ScheduleOffset=0  Type=Bulk
Pipe[1]                  : EndpointID=1  Direction=OUT  ScheduleOffset=0  Type=Bulk
Pipe[2]                  : EndpointID=2  Direction=IN   ScheduleOffset=0  Type=Bulk
Pipe[3]                  : EndpointID=2  Direction=OUT  ScheduleOffset=0  Type=Bulk


The firmware just has a loopback on endpoint 1 and same for endpoint 2 (see main.c).
I have attached a very simple Python script which puts some data through the loopback - read it for install instructions.

Goodbye COM ports!

[1]
HIDAPI library was originally developed by Alan Ott.
It is now part of the libusb project.
https://github.com/libusb/hidapi

[2]
Keywords are WinUSB, WCID, Microsoft OS Descriptors/Extended Descriptors. Some links:
https://stackoverflow.com/questions/38906880/how-to-work-with-winusb
https://github.com/pbatard/libwdi/wiki/WCID-Devices
https://www.silabs.com/community/mcu/32-bit/forum.topic.html/using_microsoft_osd-lMCD
https://docs.microsoft.com/en-us/windows-hardware/drivers/usbcon/automatic-installation-of-winusb

[3]
Guess what, also developed by Alan Ott!
https://github.com/signal11/m-stack

[4]
https://pid.codes/1209/

[5]
USB Tree View
https://www.uwe-sieber.de/english.html

[asmi]

OMG. So much trouble to get rid of COM ports, what did they do to you?
As for me, I will take zero lines of code + 1$ FT230X chip over this nonsense with Python and other crap please & thank you. My sanity is worth more than these pennies. And I can use it with any MCU or FPGA which has two free GPIO pins.

[voltsandjolts]

It's only a hassle once and now I am over the learning curve it's easy-street all the way 

[voltsandjolts]

To implement PnP WCID using Microchip libraries, see this post by NICK7 using Harmony libraries V2.02:

https://www.microchip.com/forums/FindPost/977964

[JPortici]

you actually managed to make the automatic windows driver thingy? I have  at this so many times without success..
Ah, i prefer to use libusb over windows' dlls, i havea nice wrapper around QT (derived and improved from microchip's mla examples on github) but that's another story

[voltsandjolts]

I've done a couple of projects with PIC32MM. Quite a nice cheap entry level 32-bit device family, must always check errata though.

Here's a MPLABX project for the PIC32MM0256GPM064 on the USB Curiosity Board (DM320107).
It implements WCID for PnP installation, with WinUSB as the driver. Uses M-Stack (mentioned above) for USB.
The included python script does a basic comms check, run it and you should see:

Code: [Select]

Manufacturer: Signal 11 Software LLC.
Product:      USB Stack Test Device
Serial:       12345678
Endpoint 1 loopback:  244 kbytes/sec
Endpoint 2 loopback:  246 kbytes/sec

Edit: 27DEC22 Improved comments in the python test script, no functional changes.