Yes you do need to worry about it. In your circuit, when the battery is reverse connected, the internal ESD diode in the MCU between the ADC pin and ground will be destroyed.
Simple fix! Use a 100K resistor between the battery and the ADC pin. Put another 100K resistor between ADC pin and ground. Put a 10nF or 100nF cap across this second resistor (the ADC will usually have a maximum external impedance spec of 10K or so, the cap allows the voltage at the pin to remain stable during sampling).
The 100K resistors form a potential divider, adjust them to suit your voltage reference. You may want to increase the values significantly to minimise current consumption.