I don't know the whole circuit, it has a few traps that I can see.
You are on the right track. You don't want the comparator to chatter or oscillate at the trip point. It is customary to add a hysteresis resistor ~1M-10M from LM339 output to (+) input to give positive feedback and cause it to switch solid. But your R1 1k resistor is small to allow that - how much hysteresis you need kind of depends on the battery voltage under test. I would increase your R1 to 10k with a 1M hys. resistor as a start.
Chatter will be a problem here in another way - once a battery's voltage drops and the comparator shuts off the MOSFET, the battery's voltage will creep up with no load and then circuit turns on the load and then off and then on, you see it hiccup. A latch is usually added so the MOSFET stays off once the comparator trips.
I think the MOSFET switch might not work as you intend. MTP3055VL would like at least 3.5V VGS for over 8A.
If you add a 1k-10k pullup resistor from the LM339 output to VREF_PS (+5V), the gate can go to +5V... but you do not have the source grounded, instead it goes to the load. So the source could be up a volt or two or three? In which case 5V on the gate is not enough to turn the MOSFET fully on as a switch, and it will run hot. The LED and its resistor are no good for a pullup because the LED drops a couple volts. You might want to consider a P-ch MOSFET or moving the load to the drain-side of the N-ch MOSFET if that is possible, the load not minding not being connected to the GND.