One good way to think about this is that your final Gerber data will be rounded with relatively high resolution to either an inches or mm measurement system, but your CAD system can use whatever dimensional system to define the elements you're placing on your PCBs. So, you can specify components and traces using whatever system you choose, and at the end of the day, it will all get rendered and rounded to some number of digits of accuracy to the final Gerber representation units chosen by your rendering software.
It's important to keep in mind that Gerber RS274X provides a _lot_ of resolution, far greater than any real PCB manufacturing process can live up to. So, starting with Imperial measurements and rendering metric will not result in any precision loss at all.
I use Eagle, and by switching the grid, I can use both systems on the same project at different times with full accuracy, since the internal representation of all dimensions is accurate enough to store far more precision than one could ever expect from any manufacturing system. And, that ultra-accurate floating point representation is then rendered down to a Gerber of even less resolution, which is still far more precise than the PCB that I expect to be returned to me by manufacturing.
These days, the controlling dimension for most components is Metric, and the design rule parameters for more than a few PCB fabricators are Imperial (mil) units. I recommend using Metric units when these are the controlling dimensions, and then designing the actual layout using whatever system you're most comfortable with. If you're trying to go right up to a PCB house's fab limits, and they specify these limits using mils, then lay your board out using a mil grid. If your board house uses design limits in metric units, use a metric grid or metric specified drills and traces so you can get right up to the limits of their process. If you're not interested in running right up to the process limits, choose some other grid, perhaps Imperial or metric, but scaled so that you accomplish the circuit design goals that you need to make your PCBs work the best.
I used to think that the choice of units was a critical problem, but the sane answer is to define library parts using the system that the datasheet uses, and lay out your board according to units compatible with the PCB process limits you need to work within.