We need to know what else your potential divider is connected to. Is its output just attached to the ADC, or are there any other components involved?
Do you have a capacitor between the ADC pin and ground? And is it physically as close to the ADC as it can possibly be, with the shortest possible wires?
Every time an ADC takes a sample, there's a small capacitor inside the PIC which must first be charged to the same voltage as you're looking to measure. This capacitor is part of a sample-and-hold circuit, which is required in order to ensure the ADC sees a constant voltage throughout the conversion process even if the external signal is changing.
Charging up this capacitor takes time, and during this time, the ADC input draws a little current from whatever source it's sampling. On a scope you'll see this as a small negative spike on the signal each time a sample is taken, and the size of this spike depends on the source impedance of the signal you're measuring.
With a high value pot, the spike will be larger than with a low value one.
For an input which changes as slowly as a pot, normal practice is to put a much larger capacitor (somewhere around 100nF to 1uF should do) right next to the ADC input. This capacitor supplies the brief current spike required by the sample-and-hold circuit without affecting the dc accuracy of the system.
Signals that change faster require a different approach, usually an active buffer right by the ADC - but you don't need one for this.