You use a reference voltage significantly lower than the battery voltage will ever be, and divide down the battery measurement using a resistor divider network.
For example, you could use a 1.25V reference and compare that to your battery voltage divided by two using a divider consisting of two resistors of the same value. High values like 470 kOhm would be desirable in order to not to drain the battery, but then you need to look at the input bias current spec of the comparator not to cause a significant error. You also want to add a capacitor (say 100nF) in parallel with the lower resistor to filter out any noise and interference which easily couples when you use large value resistors.
0.8V to 1.25V are usual reference voltages easily available.
There are also ICs that combine a reference and a comparator.