No; there are ferrites, i.e. compounds of metal oxides and Fe2O3, which are antiferromagnetic. These have ferromagnetic ordering but the atoms cancel each other out, giving no external magnetic field.
Sounds pretty useless, but it's not entirely so, because the key is the number of atoms contributing towards the net field. IIRC, YIG (yttrium iron garnet, a compound of Y2O3, FeO and F2O3 -- a ferrite) has a net sum so is magnetic, but fairly pitifully so (Bsat like 50mT or something, IIRC?). Fortunately, that's almost irrelevant in its typical application -- microwave resonators, circulators and etc., the signal flux is quite small indeed.
Most ferrites are of the spinel type, MeO.Fe2O3, where MeO is some metal oxide. (Spinel, the mineral, being MgAl2O4, but FeO freely substitutes for MgO, and Fe2O3 for Al2O3.) Magnetite for example is a hard magnet, not a very good one, but it is of this form (FeO.Fe2O3).
Conventional soft ferrites for power, RF and filtering are (Zn,Mn)O.Fe2O3 and (Ni,Zn)O.Fe2O3, the former having higher Bsat (up to 450mT or so) and the latter having higher resistivity and cutoff frequency.
Conventional hard ferrites are SrO.Fe2O3 and others, I forget what all exactly; they have reasonable Bsat (~0.4T) and coercivity (takes lots of amp-turns to de/magnetize them). They're nowhere near the strong magnets, but they're cheap as hell so you can just slap on some cheap mild steel pole pieces and make a powerful speaker for example.
Among strong materials: old school AlNiCo is Bsat ~ 1.5T but lower coercivity, low enough that it can't be left in free space (no pole pieces) without losing magnetization. SmCo is the second best, with Bsat ~ 1T and high coercivity. NdFeB is the current "supermagnet" with Bsat ~ 1.5T and quite high coercivity.
As for what makes a hard or soft magnet, eh, I don't recall that there's much known? Hysteresis losses in soft materials can be due to flux pinning, where a magnetic domain or pole gets stuck on an impurity or defect site; I don't know if the same mechanism can apply on an atomic scale, which might explain high-coercivity materials. Need to do more reading, it's been a while.
Tim