Does impacts,even small ones like while sliding in and out of holder, really damage the tips?
In theory: no in the short term, yes in the long one.
What damage the tip is not the small impact but the small scratch left on the tip by the impact.
Instead of the acrobatic magnetic iron holder, buy one made by a no scratching (no metal) material.
Look how Metcal, Hakko design their holders.