A 0.68uF or so capacitor in series with the battery would do it, and calibrate the zero with the 0.68uF in series with a short circuit instead of battery.
I didn't try, but my guess is, the frequency won't matter that much when measuring batteries. So, it might not be necessary to change the frequency when measuring batteries ESR. After all, this instrument produces a square wave, which means the spectrum would be more like a frequency comb anyway, and not like a single spectral line.