I would guess that the problem is the PSU's transient response—when the amplifier needs to play a loud sound, it suddenly draws a higher amount of power that the supply cannot provide fast enough, causing the voltage to droop and the sound to cut off. This could be caused either by the batteries not being able to supply enough power, or by the boost converter being too slow to respond to the sudden load increase.
I'm not sure that you can really do much… you could try adding a largish tank capacitor on the output of the boost converter (even if just to validate that this is the problem), but that will likely create other problems, and would probably just end up taking up more space than a better power supply.