That's why you strategically use star for LO and make sure current injections to signal LO are minimal.
Also meters and calibrators where it's critical have circuits for ground current cancellation to correct for offsets.
And HI guard is protected by multiple guard drivers, not just one for whole PCBA.
For parametric analyzers this issue often battled by GNDU 4-quadrant source module, that is like SMU unit, but always drives output to 0 V potential, compared to all other SMU channels. In Agilent 4142B this special SMU can sink or source 1.5A of current at 0V, and have remote sense and own guard, so you always have high quality 0V reference at the DUT device, when all other SMUs inject signal into different terminals. But this is rather extreme case and usually not needed in DMM, which is not sourcing lot of power into DUT.