Its not impossible if all parts have a sufficient supply voltage range. e.g. Microchip have many 8 bit MCUs that have a supply voltage range extending from about 2V to 5.5V which are well suited to running from 3x AA or AAA cells or a single LiPO, as the supply voltage range includes both the fully charged voltage (and the max charging voltage for the LiPO) and the commonly accepted fully discharged voltage of 2.4V (0.8V per cell) for the 3x AA or AAA batteries, and 3.0V for the LiPO.
However driving LEDs without excessive brightness change as the battery discharges can be difficult, and its also more difficult to use analog sensors etc. when you don't have a fixed supply voltage.
One viable strategy is to use a high efficiency boost regulator IC to boost to 5.0V, when running, avoiding the common variable supply voltage issues, and shut it down in sleep mode so the MCU gets Vbatt less one Schottky diode drop to maintain its state, + gate power to other parts of the circuit that don't need to be powered in sleep using P-MOSFETs.