If you like V-USB, why don't you use V-USB?
1.5Mbit/s is plenty bandwith. and USB is polled every ms (by hardware design).
As others have said before me. I suspect the largest delays are on the server side.
You might be able to pinpoint your delays easily with a logic analyser.
Get some "24MHz 8ch" boxes from Ali / Ebay / China and use them with Sigrok / Pulseview, which is a truly amazing piece of open source software.
Also:
A few days ago I stumbled into a software USB stack for ESP8266 on github.
https://github.com/cnlohr/espusbFaullt finding in an USB stack is no trivial task. the USB protocol is quite complex.
With the Logic analyser software Sigrok / Pulseview you can capture low-speed USB with EUR7 hardware from Ali / Ebay / China. Almost any board with a Cypress CY7C68013A will work with Sigrok / Pulseview.
Pulseview has a software decoder for USB to give meaning to the binary squiggles on the D+ and D- lines. It decodes T-states, Packets, headers, checksums and much more. It even flags a number of communication errors on the USB wires.