Magnetically susceptible materials (for example ferrite and YIG - Yttrium Iron Garnet) are unlike electrically susceptible materials in that they break time reversal symmetry, and so a wave traveling one direction is affected differently than the wave passing in the opposite direction.
In an electric medium, approximately speaking, the positive and negative charges of an electric dipole are separated (polarized) by an electric field. The dipoles can not respond any differently to a wave or its time reversed version because the dipole itself has no "state" that depends on time. Even a helical electrically polarizable molecule, such as dextrose (right-handed glucose) can not break time symmetry, because while the shape of the molecule allows charges to move helically along its length, which produces a different in retardance between circular polarizations, it still responds based on the chirality of the molecule which does not change with time or direction.
On the other hand, a magnetic dipole breaks time symmetry. If you imaging a magnetic dipole, it's a current loop. That current loop (for example the spin of an electron or its motion around a nucleus) has a definite direction the current flows around the loop. If you reverse time, that direction reverses as well. So the magnetic dipole responds different to a circular polarization which turns in the same direction that the current is flow in the loop, vs. the polarization which turns in the opposite direction, tending to be magnetically polarized more by when they rotate in the same direction, resulting in increase retardance of that circular polarization as compared to the other.
Also, if you imagine the magnetic dipole being viewed from one side or the other, the direction the current appears to flow reverses, so that a linear polarized wave incident on the magnetic dipole for one side or the other, which is a superposition of left and right circular polarization, is going to have its two constituent polarizations retarded differently depending on direction. The linear polarizers on an isolator can be set up to allow the wave through if it travels in one direction through the magnetic medium and block it if it travels back the other way.
Basically, a material like ferrite or YIG is a bunch of electron spins that have been aligned due to paramagnetism or ferromagnetism, and each electron spin acts like a tiny current loop with a direction of current flow given by the right hand rule. You can imagine classically that each electron is like a little spinning ball of charge, however, quantum spin is simply an inherent property of the electron and has no physical manifestation except given by the effects of symmetry (Pauli exclusion principle) and electromagnetism.
In classical electromagnetism, the loss of time reversal symmetry is because the magnetic field depends on the first derivative of time, so that if time is reversed, the derivative is multiplied by -1. Or in other words, if you reverse time, the direction of a current reverses and it becomes the opposite current. This loss of time reversal symmetry because the medium has a net magnetic polarization is why it is used in an isolator/circulator.