All depends what's stuck in said sponge.
If some abrasive is added, the answer is absolutely yes. Lapping plates for example are made of gray cast iron, a somewhat spongelike metal where the voids are a few percent of graphite flake (which is so soft as to be practically irrelevant to abrasive particles). In addition, the iron (ferrite*) phase is soft, tending to be dented by, and thus trapping, abrasive particles. In both cases, they can wear against a harder surface.
*Metallurgical name, not the ceramic compound (XFe2O4).
This often happens accidentally, in practice, when some grit or shavings get stuck under an oiler pad, or soft metal bearing, or etc., and the substrate gets gouged out (or worse).
Tim