For small batteries/cells, I do this with a bench top DC supply and a resistor.
The idea is to sink enough current in the resistor that the DC supply never gets "back fed" from the charger (the DC supply always has to set the voltage across the resistor, not the charger).
E.g. say your charger is able to output 100mA and the lowest battery voltage you want to test is 2.8V - so you choose a resistor that is able to sink 100mA at 2.8V, or 28 ohms max.
Test method:
1. Connect your benchtop DC supply to the resistor and turn it on. The supply sets the "battery" voltage - so if you want to test with a battery voltage of 3.7 V, you just dial that in on the supply.
2. Connect the 100mA charger. What happens now is that the DC supply will reduce its current by 100mA, to make up for the current from the charger and keep the voltage at 3.7V
Now, you can literally just dial the bench supply to any setting between 2.8V and 4.2V (the battery max...) and see how the charger behaves.
All for the price of a resistor you probably already have in your junk box!

[note: this is the same idea as @Berni above]