Rpi Bitbanged USB-HID Host

[PhazeHenot]

I am working on a project using the SMI (Secondary Memory Interface) peripheral found on most raspberry pi models (Pre Pi 5) to implement a bit-banged USB Low Speed HID Host. Primarily intended for using USB keyboards.

I have gotten decently far in having successfully implemented NRZI packet encoding/decoding, data transmission, device enumeration, and polling, albeit only with my test keyboard thus far (see attached video below).

I am having an issue however where due to the software processing of the data received packets often come in corrupted. The program is not fast enough to analyze a packet before the ACK timeout elapses and so simply heuristically ACKs every received data packet and re-requests if the data is corrupted. This is an acceptable tradeoff for a keyboard driver as no important data is likely to be lost in transit.

Where this becomes an issue however is that the USB-HID protocol defines an Idle interval, wherein the keyboard will respond to further polling requests with a NAK packet as opposed to a full key report in order to save bandwidth if the pressed keys have not changed, meaning that if the initial packet gets corrupted the program has no way of knowing which keys are pressed until this idle period expires. The HID Spec mandates a command "set_idle" where this interval can be set but it seems a large number of USB keyboards do not implement this correctly including the devices I am testing with which have an indefinite idle interval and so will never send another report untill the pressed keys are changed.

There is also supposedly another option the "get_report" command which is supposed to allow requesting a report via the control endpoint, but my test keyboard seems to incorrectly implement this as well.

If any of y'all have any ideas on how I can force a full report from the keyboard in the event of a corrupted packet I am all ears. I have attached a repository link and a (poorly shot) showcase video below.

EDIT: I have largely fixed the error problem. My RX buffer parsing function was incredibly naive in assuming that signal noise would only mess with the contents of each NRZI run and not the length. I have since amended it to handle each run in one go and extrapolate how many bits the length corresponds to instead of taking a certain number of samples and assuming they align to bit boundaries. This has reduced the error rate to practically zero and it now enumerates and polls the test keyboard perfectly. Now all I have to do is figure out why polling is not working on other keyboards.

github.com/StrayerRussell/rpi_smi_usb
https://youtu.be/Oza2_v8EOi8

[magic]

I am having an issue however where due to the software processing of the data received packets often come in corrupted. The program is not fast enough to analyze a packet before the ACK timeout elapses and so simply heuristically ACKs every received data packet and re-requests if the data is corrupted.

Well, you were not supposed to do that, I guess

Where this becomes an issue however is that the USB-HID protocol defines an Idle interval, wherein the keyboard will respond to further polling requests with a NAK packet as opposed to a full key report in order to save bandwidth if the pressed keys have not changed, meaning that if the initial packet gets corrupted the program has no way of knowing which keys are pressed until this idle period expires. The HID Spec mandates a command "set_idle" where this interval can be set but it seems a large number of USB keyboards do not implement this correctly including the devices I am testing with which have an indefinite idle interval and so will never send another report untill the pressed keys are changed.

Maybe try issuing the clear_feature(endpoint_halt) request and poll again (don't forget to reset your toggle bit). In general, that's what you are supposed to do when you detect that protocol state got screwed up somehow. How your keyboard will react to it I don't know.

[NiHaoMike]

Might it be easier to use a microcontroller with USB host as a USB to SPI bridge?

[dobsonr741]

I would not expect the USB host to be consistent in Linux user space. Considered going bare metal? On https://github.com/rsta2/circle ?

[langwadt]

I would not expect the USB host to be consistent in Linux user space. Considered going bare metal? On https://github.com/rsta2/circle ?

or https://github.com/ultibohub

[dobsonr741]

Oh, the pascal people. Good memories. Brought up turbo pascal the other day in DosBox on Apple silicon Mac. How long we came.

[PhazeHenot]

Might it be easier to use a microcontroller with USB host as a USB to SPI bridge?

Yes but less informative. The main reason for doing this project was to gain an understanding of USB at a fairly low level, and also because it seemed just barely within the realm of possibility.

[PhazeHenot]

I would not expect the USB host to be consistent in Linux user space. Considered going bare metal? On https://github.com/rsta2/circle ?

The intention is to make the final version a kernel module. Just doing the proof of concept in User-space for ease of troubleshooting. Once I get the keyboard working semi reliably I plan to move everything to kernel-space and add way more error checking/fault recovery.

[dobsonr741]

Even in kernel space the timing is unpredictable. A magnitude better on bare metal, like Circle. If you are not doing anything else.

Just curios, what happens next after you bring it up error free in the Linux kernel?

[aeg]

I am having an issue however where due to the software processing of the data received packets often come in corrupted.

You need to fix whatever's causing this, corruption of USB packets should be rare, especially at low speed.

The program is not fast enough to analyze a packet before the ACK timeout elapses and so simply heuristically ACKs every received data packet and re-requests if the data is corrupted.

The "right" answer is probably "don't do this", but if you're doing this out of sheer perversity... how about NAKing every received data packet and letting the device keep re-sending until the host has finished analyzing the first one?

[amyk]

You need hard realtime to do this, which a general-purpose OS is never going to provide.

[Berni]

That is a very weird thing to want to do in Linux. What is the purpose of this?

Most things that run Linux already have a USB port. Even if an extra USB host port is needed it is easier to add a chip such as a SPI to USB chip and run it with an existing kernel driver.

As others said above, doing this with the regular linux scheduler running the show is never going to be reliable.While it can offer pretty good timing in ideal scenarios, there is always occasional blips in latency when something in the kernel takes longer than usual. That gets even worse once the system is under heavy load and even worse when memory starts getting paged in and out. It could work with a special realtime focused kernel or by making your own real time bubble inside the existing kernel by setting up a hardware timer to fire off a high priority interrupt that gets handled by your own code.

If only reading standard HID input devices is the goal, then the easiest solution might even just be a small MCU with USB host capability, then talk back to the SBC over whatever other protocol is convenient.

[PhazeHenot]

Even in kernel space the timing is unpredictable. A magnitude better on bare metal, like Circle. If you are not doing anything else.

Just curios, what happens next after you bring it up error free in the Linux kernel?

Main point of this was just to see if I could and gain a better understanding of USB/DMA and kernel programming. Provided that it winds up being semi reliable it might be fun to use this to make a 104 button macro-pad with a PI zero and very minimal external components.

[PhazeHenot]

That is a very weird thing to want to do in Linux. What is the purpose of this?

Most things that run Linux already have a USB port. Even if an extra USB host port is needed it is easier to add a chip such as a SPI to USB chip and run it with an existing kernel driver.

As others said above, doing this with the regular linux scheduler running the show is never going to be reliable.While it can offer pretty good timing in ideal scenarios, there is always occasional blips in latency when something in the kernel takes longer than usual. That gets even worse once the system is under heavy load and even worse when memory starts getting paged in and out. It could work with a special realtime focused kernel or by making your own real time bubble inside the existing kernel by setting up a hardware timer to fire off a high priority interrupt that gets handled by your own code.

If only reading standard HID input devices is the goal, then the easiest solution might even just be a small MCU with USB host capability, then talk back to the SBC over whatever other protocol is convenient.

The purpose is mostly learning. about USB fundamentals, low level hardware interfacing from Linux, kernel driver development and such. I have no doubt that if I was trying to make a product it would be far more reliable to have a microcontroller acting as an interface to the keyboard but I wanted to see if I could get it working with only the native resources of the Pi instead of requiring additional external components.

That being said the kernel is having to do very little actual real-time work. as the SMI peripheral of the PI allows configuring incredibly precise timing intervals and can clock data in/out at up to 50 M/s into FIFOs which are serviced by DMA. The main thing the CPU has to do is prepare the data and initiate the SMI transfer which then proceeds along according to its programmed clock rate. This is still leagues less reliable than dedicated hardware, but handles most of the time critical components (precise timing of data out and clocking data in) with the only hiccup thus far being the corruption of polling data. Even device enumeration has been fairly reliable up to this point.