USB is not great, but it's not as horrible as people make it sound. Yes, it has unfortunate performance characteristics (like USB1.1's "sending a DATA packet over and over just to receive a NAK for it every time"), but once you see the limitations as trade-offs (ahem), it's something workable. Not great. Don't get me wrong. I'd still not use it for chip-to-chip connectivity.
Even enumeration isn't that complicated. Implement GET_DESCRIPTOR (simple, just return a fixed string based on wValue/wIndex), SET_CONFIGURATION (dummy), GET_STATUS (dummy), SET_ADDRESS (just poke your hardware's address register) and be done. It won't be standard compliant, but it gets you enumerated. That's ~50 lines of non-dense code.
And then, yeah, just use bulk transfers.
What makes USB really frustrating, in my mind, are two things: a.) you don't need to understand it in order to use it, and b.) USB is cheap.
For a.), it allows you to go with the black box approach (take vendor code, compile, use, throw it against the wall because it doesn't fully work and hangs every two hours) for quite a long time. You can develop USB devices these days without understanding which USB packets exist. That's different from, say, TCP/IP, where Ethereal is your friend and you get full insight down to ethernet level with a few mouse clicks. Or a serial port, where, to be honest, not many corner cases can happen anyway.
But with USB it's different. You _need_ to understand what's going on on the wire, otherwise the first issue you run into will turn into a heisenbugfestmess. The issue is that it's not easy to get there. usbmon and friends are not telling you what's going on on the wire, but instead what _should_ go on on the wire. I was scared of USB until I bought a (even back then overpriced) LeCroy analyzer from the BenQ germany auction. But it was the first time I could actually make sense out of those dreaded kernel messages a la "Device does not accept address" etc., and get a hint in debugging them other than just cargo-cult programming. USB analyzers thankfully got less expensive.
For b.), the issue is that USB is so mass-market, you can't trust anything anymore. Your hub? marcan will happily tell you horror stories. Cables? Don't get me started. Cheap usb controllers that just don't respond to certain commands, and an equal number of ugly workarounds in the kernel drivers? Check.