For 4xAA, I guess it depends on what voltage range your device can run on:
New alkalines are around 1.6V and fully discharged, they can go below 1V.
If it e.g. cannot go below 5V, you will waste around half the capacity.
And a step-up module may have some (significant) quiescent current.