Out of curiosity, the human body model for capacitance is 100 pF. Is that enough to cause the damage we're talking about?
Short answer, yes.
Longer answer: the input capacitance for a CMOS logic gate is approximately 5 pf, or 5% of the capacitance of the human body. If the chip was stabilized at a static potential of, say, 1 kV with respect to ground, and you touched one pin of it (you always touch one pin before you touch the others), then the 5 pf capacitor dissipating into the 100 pf capacitor of your body is going to bring the voltage of your body only up a little bit, roughly 5% of the original 1 kV. That means that, after equalization, the input pin is at a potential very close to ground, while the rest of the chip is still at its original 1000V. In other words, most of the static voltage that was on the chip is now present across the thin dielectric of the input transistor. Will the dielectric break down under that high voltage?
This is pretty nearly the same as what happens if you had neither a grounding strap nor a static mat, and you picked up a chip that was at near ground potential while you had a static charge on your body (except the sign of the charge may be reversed, a bookkeeping issue only).
OK, it could be that a person walking around has more opportunity to build up a significant static charge than an object laying on the bench. By that logic, maybe the wrist strap might be a bit more important than the antistatic mat. But they're both important.
Another thing to note -- it doesn't make a whole lot of difference whether things are at ground potential or not -- what matters is that they're all at the SAME potential. So it's more important to see to it that there's a current path between your wrist and the work surface than to see to it that there's a path between your wrist and earth ground.