I've been thinking a little on this, and come the the conclusion that unless you tell us what signal you are filtering and what you want to achieve it is very difficult to offer any advice outside of the basic implementation code.
Maybe the input signal has only very low frequency components (e.g. monitoring a battery voltage), in that case taking four samples very quickly and averaging them is a valid way to remove random conversion noise between samples. If the random noise has the correct statistical properties it might even be able to add a bit or two of effective resolution.
However, if the ADC's input signal contains any high frequency signals (relative to output the sample rate), then averaging three out of four samples is bad. You don't see the difference between (50, 50, 50, 50) or (100, 0, 100, 0), (100, 100, 0, 0), (67,67,66,0) or (200, 0, 0, 0) as they all average out to 50. Because of this it really depends on signal conditioning (analogue filtering) before it goes into the ADC.
One easy test would be to print out a dozen samples - if the ADC reading is stable in the LSB then there there is no more information that can be gained through averaging it.