What your after is not really an average, but a statistical filter, like mode or median, in reality all 3 of these approaches are more computationally expensive because you need to sort them,
What you may want to try is a limited difference average, where outside of the window the values get scaled down to have less influence, such an approach is only slightly more expensive than an average, and only needs slightly more ram than a single value average.