Or just put a (few) diodes or an LED in series with the LiPo.
You do have to make sure the voltage drop is adequate over the whole power range (idle and sampling).
Voltage regulators always need a bit of current for themselves, and you have to look good at the specifications to verify the one you use does not need to much. Some go up to several milli amps.
LiPo's also need a good cutoff as protection agaist over discharging.
LiFePo may also be a good solution, It has a bit lower output voltage, but I think it's also more robust.
Yet another option is to use LIR2032. Same size as CR2032, but rechargeable.
Edit:
I had a click on that amazon link, and according to the description it already uses LiPo.
Do you get some commission for page visits on amazon or something like that?