If you attenuate the output and reduce the gain accordingly instead of the input, you 1) reduce the amount of gain needed, 2) provide a fixed input impedance, 3) make the load more resistive (although HPs are already mostly resistive), and 4) makes the input offset voltage non critical: little gain is applied, and it's divided down on the output by the volume control, so output can be DC direct. It's only power amplifiers where output attenuation isn't possible, but for µW type loads like HPs it's the go-to. If you attenuate first you push the signal down further in the noise floor, only to pull it back up again with gain which invariably pulls up noise with it. (See Friis' formula.)
Here's a super simple HP amplifier, but you'd be surprised how good it sounds with a monitor-like headphone like the AKG K102 and a quality source. I sell a slight variation on it, purely based on word of mouth, to some very picky people.
Thumb:
Full PNG:
http://www.rockgarden.net/download/eevblog/hpa.pngKiCad v5 schematic:
http://www.rockgarden.net/download/eevblog/hpa.schEnjoy!