estimate the average power consumption of that device and the peak current (both at 1.8v, the minimum voltagE0, then maybe pick a buck switching regulator with small enough quiescent current and enough current output for your needs, and configure it to track the discharge rate of the battery ( 4.2v full to 3.6v discharged ... so 2.4v to 1.8v ... basically half the input voltage)
High frequency dc-dc regulators would use very little space, it would be just an issue of balancing the maximum frequency (for tiny inductor and capacitors) vs efficiency at some output current range.
For example see
TPS63030 which uses only a 1.5uH and a bunch of tiny ceramic capacitors and can output up to 800mA of current.
Or see NCP1406 / NCV1406 for lower frequency.
Or see
TPS62240 for 300mA maximum current.
Since you want exactly half the voltage (1.8v .. 2.4v for a battery that would have 3.6v to 4.2v) ... for low currents (like 50-100mA), you could look into voltage doubler chips which can also be configured to output half the input voltage.
For example,
For maximum 40mA of current , see
LM2665M6For maximum 60mA of current, you could try
TPS60403For max. 100mA, you have
LM2660MXfor max 200mA you have
LM2662MXSee the datasheets included on the pages linked above.