Speakers don't have a flat, resistive 8 or 4 Ohm impedance at all frequencies. Does anyone know whether passive crossover networks are designed to take this into account? If so, then changing the speaker, will affect the frequency response.
I think you should try to stick to the same kind of speaker, both physically and mechanically to stand the highest chance of it giving good sound quality.