A multimeter probably won't draw 25mA, though.
Absolutely;, Even a very cheap contemporary DVM, like that often given away for free by Harbor Freight has an input impedance of 10 MΩ or better.
I assumed (very often an error) the OP would understand he'd need to use some external load or a real battery tester which if properly engineered would apply appropriate load to the test cell--something often not understood.
In fact a particular inexpensive (which does not always mean bad) digital battery tester offered on eBay was trashed by more than a few reviewers because of it's "inaccuracy". All to a man reported how they had used the tester to test a cell, and then measured the same cell using a DVM--lo and behold, the tester consistently read 0.2 to 0.5 V lower than their, of course perfect, DVMs, proving beyond doubt (in these reviewer's minds) the "inaccuracy" of the tester.
I have that same tester. when testing a 1.5 V cell it places a 47 Ω load (32 mA) on the DUT which of course pulls the no-load voltage down by 0.2 to 0.5 V depending on the condition of the cell.