There are advantages to using an active mixer, but I mix little bits of signals here and there passively all the time, and it works fine. The electrons cooperate; they just do their job!
Headphone amplifiers tend to be all over the map in terms of their output levels and impedances, as do headphones with their inputs, so the best way to decide how to build the dividers would be to experiment a bit, or to get downright methodical and measure some stuff. I see some specific values in your circuit, but I don't know where they came from, so...
If you can get the same signal from the two sources, at their "normal" output levels, then figure out how much gain reduction you need from the loud one to make it the same level as the quiet one. This could be done with a meter that reads low AC values, or some other piece of equipment you have that can give you reasonable metering of some kind. This comparison should be expressed in terms of a ratio, so you can work with it easily when you're figuring out the voltage divider. If you method of checking the gain reduction gives you decibels as the answer, then the ratio = 10^(db_delta / 20). (If you find that 6dB = 2x, you're doing the math right.)
So if I was doing this, I think I'd start by trying to get that data, and then decide if I even need a pad switch. If so... if one level is just way too much stronger, I think I'd add a -20dB pad (or whatever nice round number!) and let it go at that. And the easiest way to do that would be just use the volume pot as the bottom of the voltage divider, and add a resistor in line with the input to it, with a switch across it to short it out to disable the pad. That eliminates more of your (rather large!) collection of resistors in the circuit, so that's a Good Thing.
Then there's the panning and other passive mixer bits. I can't think of a real-world scenario where that would be necessary, but I don't know what you're up to, so I'm only working from my own experiences. For me, the simple thing would be to just use an isolation resistor to go from the volume pot wiper to the output, and that's that.
But if you really do need panning, and if you take a step back and look at the circuit, you got another divider between the two outputs, with the two outer thirds fixed in place, and an adjustment across the inner third. In other words, you can go from 10k : 20k to 20k : 10k, with 15k : 15k in the middle. So the voltage adjustment for one side would be from 0.67 down to 0.33 of the input, which is a variation of (20 * log (0.67 / 0.33)) 6dB. Not very much.
I'm not sure how to figure out what all the resistances should be, because I don't know the output impedance of the source or the input impedance of the next stage. 10k seems high as a ballpark figure; I think I might use 1k everywhere... 1k pots, 1k isolation resistors, and that's that. If we can live without the pan, that's the optional pad with one resistor and switch each, the volume pot, and the two isolation resistors. Low parts count! That's good, because a resistor's job is to throw away the voltage you've worked so hard to produce.
I'd build a simple prototype and try it, before making a fancy box with connectors and labels stuck on it.