Are you sure about putting 24V on a USB connector? That could lead to smoke, if something which expects 5V is plugged in.
Switching the ground can also cause problems. If the power supply to your circuit is earth ground and you use it to switch something who's 0V is also connected to earth ground, then it won't switch it off, because the current will find a path throuh earth ground, bypassing the switching transistor.
Finally, have you considered using a small microcontroller? As well as being cheaper and simpler, the internal RC oscillator will be much more accurate and stable, than discrete components.