When I got my 4503, I had very similar problems and the resistances ranges were useless, and it was all down to the battery leaking just like yours has. Personally I'd 100% replace the battery with the same, but I'd take it out of circuit first and don't worry about losing the calibration data, it is fairly easy but time-consuming to do a self calibration and adjustment via the front panel, just follow the instructions in the service manual. You don't need the equipment they tell you to use, all you need is another meter that you know is in spec and some reasonable stable voltage, resistance and current references, and then you can tweak the 4503 readings by following the instructions until you can get readings that match the known meters.
Once you have the battery removed, thoroughly scrub the board on both sides many times with some 70% Isopropyl and dry carefully. Repeat this many times especially in the area around the battery location and the corner quarter of the board where the Craddock precision resister network is located needs extra attention as this area is extremely high impedance as its the meter inputs section and any contamination in this area will screw the meter readings. Also, I'd do a full recap of the board as even though they don't look bad, they are the original caps and many of mine were found to be dry and way out of spec.
Those boiled sweet looking polyester caps and the resistor by the battery are showing badly compromised leads and solder joints, which mine did as well. I'd clean them up, de-solder them and check them, and either replace them or if they are OK and you can clean the legs up OK enough to be able to tin them. Pay particular attention to holes that the leads of the caps pass through and make sure that are clean and the via's, pads and connecting traces are not compromised and repair as needed. Only when you have cleaned the board loads of times, fit the replacement battery, and then test and the readings seem to be more stable, do a recalibration via the front panel and then hopefully you will have fixed the meter, they are worth the extra work.