Hi,
I would suggest that you look at acquiring a drill press. While the ideal tool will be high speed, precision, rigid, well balanced, and designed especially for PCBs, I have found you can get quite good results with the drill presses you can get for around AUD120 in hardware shops. These will usually go up to about 3600RPM, which is a bit slow, but OK if you are not in a hurry. Make sure that the chuck runs true and will hold the drills you are intending to use. You can get mini 'adapter' chucks, but in my experience the drill bit ends up all over the place with these. A better solution is to use a bit with an expanded (~3mm) shank.
You will also want to get some carbide drills - normal HSS ones will start to go blunt after about 10 holes, and blunt drills make a real mess of the PCB. You need to take a bit of care with the carbide bits as they are quite brittle, and somewhat expensive (~AUD5 each), but you will get much better results.
Set up the press so the drill is only about 1mm above the board, and you will be able to drill holes with a light touch on the handle. Good lighting is required - eg a 50W halogen lamp right up close. It is also more convenient if you have an air blower to remove the swarf. I use a 12V air mattress inflator for this.
Try to resist the urge to automate the whole process by building a CNC drilling robot. I played around with this for years, the accuracy I achieved was only just acceptable, and I don't think I really saved any time with it. But it was a fun project!