Well, I dont know, but isolated supplies are not totaly isolated, they have AC leakage through capacitance if output side is not grounded. Easy way to look for such things is with multimeter. You can measure AC voltage between your USB ground and speakers input jack. But even in cases of leakage, the current caused is usually very low, so it is not likely to damage things easy, but who knows. Measure and maybe it will get more clear.