Buy a cheap jewellers loupe magnifier off ebay, even the sub £2 ones are OK.
The engraving bit you're using is a good choice (I use them myself), but you absolutely need to get it spinning true - i.e., with no observable runout. With the spindle running, get up really close to the bit and look at it through the loupe. If you can see any shadowing around the tip, stop the spindle and reseat the bit. Repeat as necessary until the tip appears needle sharp with no blurred edge.
You can also use the loupe to check if the PCB is totally flat. With the spindle running, just travel to the four corners, and incrementally lower the bit until the tip just touches the surface and leaves a small mark. If the machine is up to it, you should be able to get a resolution of <50um. Either shim up the low corners or compensate in software. (A discrepency of 50-100um across the PCB is usually OK, any more will require adjustment.)
For cutting depth you want to be starting at 50-100um, which is 0.002-0.004 inches. Obviously go a little deeper if necessary.
For cutting speed, the spindle should be as fast as possible, and the feed rate about 10-30mm/min. If you go too fast on these hobby machines the cutting pressure will deflect the bit and leave a poor cut. Slower is better.
With a bit of practice you should be able to cut tracks running between 0.1" IC pins, and SMD boards using 0805 components.