If the midpoint of a 50k 'logish' pot is 10k, then in that position the pot has a gain of -14dB (You can play with this a bit by adding a shunt resistor to one or the other end of the pot).
22k is way too low when using a 50k pot (Which has a worst case (50% resistance) source impedance of 12.5k, you solve this with a buffer opamp after the pot, cap coupled from the pot wiper (or use a fet input opamp).
You follow the buffer with your summing resistor and summing amp in the usual way, but you can now use much lower values so that you get a sane noise floor and whatever gain you need (I would suggest 14dB as a starting point!).
You might also with to explore Bauxandall volume controls, lets you use a linear pot.