MCU ADC is similar to the multimeter in that it "loads" the signal. Just that the load is a capacitance, not a resistor. Meaning, it consumes a current peak when it samples. ADC basically connects a tiny sampling capacitor into the input for a short time when triggered to do so.
With 2Mohm resistors, charging the ADC sampling capacitor is slow, but the ADC sampling time is limited (maybe fixed, maybe configurable, but limited anyway), thus it does not fully charge to the real voltage, causing wrong result.
If you don't have to sample all the time, but e.g. once a second suffices, then simple solution is a capacitor between the ADC input pin and GND. The ADC pulls its required charge from this capacitor and gets it quickly. Then, between the samplings, there is plenty of time to slowly charge that larger capacitor through your 2M resistors. This capacitor should be e.g. at least 1000 times bigger than the sampling capacitor. 1uF ceramic is a good idea.
Now if you have to run continuous sampling you have no other options than to provide continuously low impedance for the ADC and you can do it by significantly reducing the divider values, or by buffering the divider output by an opamp circuit.
But for your use case, capacitor will do fine, it's not like battery voltage changes quickly anyway. Just don't enable automagic continuous sampling from the MCU, instead trigger sampling by software or a timer.
It's still a good idea to reduce the resistor values somewhat if accuracy is important. There can be leakage currents in ADC pins, the capacitor can leak a bit, surface contamination on the PCB traces, especially flux residues from soldering. These would be in the gigaohm range but still measurable effect when paralleled with 2M if you are aiming for better than 1% accuracy or so.
But generally the idea of using large resistor values especially when permanently connected to the li-ion cell is good one. This way you minimize current consumption during off-state and prevent overdischarging and killing the li-ion cell even during long period of non-use (sitting on the shelf).