Measure across flats, thats how it is done in datasheets where you buy nuts from. Also note that hex nuts have the corners of the hexagon somewhat rounded off, so a corner to corner measurement would end up giving too small a hexagon to encompass the actual nut if used. Recesses to take hex nuts should be shaped according to the hexagon with flats measured (or read of the datasheet for that nut type, they are standard for the M series metric thread sized nuts), then offset by 0.1 to 0.3mm to widen it. The amount of offsetting epending on printer accuracy and whether you want a tight fit (struggle to force the nut in) or a looser one (nut at risk of falling out until screw tightened in to it and maybe at risk of turning in the hex recess if sufficient torque is applied to the screw). 0.15mm to 0.2mm (added on each side of the flats, so 5mm across flats becomes a 5.3 to 5.4mm hole across flats) usually gives the bext balance of tolerance considerations.