I'm noticing some confusion here between asynchronous USB transfers, that don't exist, and isochronous transfers. As far USB is concerned, there are two types of transfers for massive data: bulk and isochonous. Both have CRC verification, but only bulk guarantees correction (i.e.: data is delivered, if not, rinse and repeat until it is). Other types of transfer exist for control and signaling: control and interrupt transfers. Hope this little bit of information is useful.
I apologize — when I wrote above that USB audio uses asynchronous transfers, I should have been clearer and said that it uses isochronous transfers that in turn use one of three synchronization modes, which are asynchronous, synchronous and adaptive.
Thanks for clearing that up. The asynchronous/synchronous/adaptive business seem to be a specificity of the audio class devices. Yes, the transfers would still be isochronous.
Anyway, I still bet that this has not to do with jitter. What the OP is experiencing is a ground loop.
Kind regards, Samuel Lourenço