A dumb question but I don't see the reason why... Most headphone outputs have a resistor in series with the output ~100-220 ohms and sound fine.
But take an op amp that has a built in 75 ohm resistor in series with the output and it just doesn't so tiny...
Is it the feedback loop? because the feedback is taken from the headphone side instead of the amp side? but I would think that would instead boost the bass since that creates a high pass filter (in the negative feedback circuit)