Funny enough I bought a similar cheap USB audio device, and improved it simply by cutting the USB power trace near the plud, and inserting a LC filter, using some salvaged SMD ferrite filters and some 22uF 10v tantalum capacitors. Got rid of a lot of the noise, as most is coming in via the power line. Mght be wotrth doing it to yours as well, though you have a lot more options for inductor because you are not needing it to fit back in the original plastic housing, though you will want a inductor with high loss over high inductance, to avoid ringing, so probably best to use a few dozen turns of wire through another lossy ferrite bead, and solder the capacitor direct on the old USB plug mountings.