I'm not sure you have any problem at all, and would suggest not opening up the probe!
The DC level control and Balance are for different things. The DC level control is for shifting the trace on the scope, while the Balance control is for zeroing out the probe offset. The probe is completely disconnected in "CAL DC LEVEL" mode, and balance will have no effect here.
Zero the trace using the DC level in "CAL DC LEVEL" mode, then switch to DC mode with no conductor in the probe, then degauss, then use balance to re-zero. Ideally there should be no trace shift between "DC" and "CAL DC LEVEL" mode with no conductor present in the probe. Do all this after a substantial warm-up period, there's a reasonable amount of drift!
If balance has no effect in "DC" mode then yes, you may have an issue.
These probe amplifiers are a bit strange by today's standards - while they are not a true scope plug-in module, they do a lot of the vertical scale/offset stuff normally found in the scope. Ideally you'd just have 2-3 ranges and no DC level control, and do all that stuff with the scope controls rather than the amplifier (this is how the newer amplifiers work I think). I may try and build my own amp one day to go with my probe along these lines (there is an old Jim Williams linear tech app note about doing some of this).