You're mixing up things, so let's get a couple of points clear:
Audio power amplifiers (headphone, loudspeaker, whatever) are voltage sources, and will deliver output voltage = input voltage times X.
This means, that for a certain output voltage, the only thing limiting the delivered output power is the impedance of your phones/loudspeaker.
If you supply 1 Vrms to a 32 ohm phone, it will dissipate 31 mW (and hopefully convert most of that to audio). If you supply the same to a 600 ohm phone, it's 1.7 mW. If it's an 8 ohm speaker, it's 125 mW.
The amplifier will at some point protect itself by limiting output current (eg, when looking into a 1 ohm load), but that's no longer normal operation.
The sensitivity of your phones is a completely different discussion.