When I roll my own boards I fully expect the fit and finish to be a little ugly. If you want pretty, farm it out! That said, if I'm working on building a prototype I tend to use the 3/64" bit you find in the common 7 piece dremel index. It's a little big but it's not horrible at least for the 1/2w resistors and more common TH caps I use. For 1/4 or 1/8w components you might want to go down to the 1/32" bit depending on the lead size. If you want the hole size bang on exactly right, then you'll probably want to invest in a set of precision bits and a drill press.
Here's my personal slick trick for those "off the shelf" bits. I use that broken bit you speak of! When I secure it in the chuck there's only a small nub poking out and it doesn't flex or wobble even with a fair amount of downward pressure. It's just long enough to get through about 3-4 mm worth of FR-4 stock.