Measuring on FR4 is inaccurate because of the FR4 and probably not because of your calipers. Unless you don't know how to use the calipers.
Hole diameter: (outside dim. - inside dim.) /2 = hole diameter. Because FR4 is so inaccurate (relative to measurements on machined surfaces), I just use a drill to estimate hole diameters. With Imperial, I have a set of number, inch, and letter drills. A similar set for metric should work as well. Since most are probably drilled with metric drills, a good set of metric should get you close.
EDIT: Be sure to use the drill shank as the plug gauge, not the cutting end. Measure the shank to be sure it is close enough to the labeled diameter of the drill.