I use mils for layouts, both for schematic and PCB. Many board houses (maybe all) convert drill diameters to metric. A typical drill will bore a hole to something like ±0.001" to 0.002" (0.002" ≈ 0.05 mm). As Eagle (and probably most others) convert to 6 decimal places, whether one uses mils or metric really doesn't matter in practice; although, the displayed values may seem strange. Metric drills are probably used by the board houses, and I would defy anyone to measure a whole in FR4 after plating and tell me whether it was drilled by equally precise metric or roughly equivalent imperial drills or visa versa.
In other words, use what you're used to and make your component footprints to match. For manufacturer recommended footprints, you will usually see them in both metric and imperial units. If you actually calculate them, you will see there is rounding. A potential problem may occur when routing both metric and imperial components on the same PCB. Many years ago, EAGLE would recognize an intended connection but consider it an overlap. It fixed that problem. More recent editions automatically snap to the center of a pad if you are anywhere near close. I think that change occurred on or before version 6.0. (Sorry, I don't remember when.)
As a practical matter, I keep a function key (F5) assigned to swap from imperial to metric grid with a single keystroke. The alternate grid, which is usually finer but doesn't need to be, is fine enough such that swapping back and forth makes no discernable difference nor does it result in DRG errors.