The oxide layer is pretty transparent at the few nanometres of thickness it has, and covers the whole surface. The only way to get a clean, pure surface, is to use a chemical cleaner, and it doesn't last very long. This is one reason that despite the much lower cost of aluminium, copper is still preferred for most electrical local supply installations (it is used for grid and supergrid, but that's a different story) because it's very troublesome to get a proper termination; you can't just make off an armoured cable and shove the ends into a connector block or crimp, you have to use special materials and techniques, and the connection is still more likely to cause problems over time.
When you stick your DMM probe on an exposed bit of aluminium frame or bracket, your probe tip breaks through the oxide layer. When inserts are crimped into an Al part, the electrical connection relies on displacing material to ensure a mating surface that is not exposed to atmosphere. It's still just a matter of time before they fail, and it won't necessarily be visually obvious, unlike with say, a rusty steel or iron part.