I think the analog front end is the hardest part. Precision and high impedance/ low leakage stuff. Read some service manuals for older bench DMMs, e.g. HP 3455A, HP 3456A, Keithley 19x, Fluke 8840A. And look into the DMMs itself if you can. See how they build the front end, which components they used, when they used teflon, how the guarding is done, etc.
The ADC is one of the easiest part. Just buy a suitable one. The LTC2400 isn't good for this application. You want one which measures positive and negative voltages.
The analog part and ADC should be galvanic isolated from the rest. And the voltage and current inputs should isolated too if you want to measure both at the same time. A shared common for both is problematic due the voltage drop of the leads.
Another problem is calibration and performance verification.
A DIY 5.5 digits DMM won't be cheaper than buying a used one and probably a new one too.