It can happen. I did a project last year where RJ45s would've been too bulky, and unsuitable for the environment (lots of little industrial modules in a chain). So we used small headers (housed with snap retention).
And remember, signal quality is basically irrelevant, I mean look at RJ45 to begin with, they aren't even using the right pins!
I would look for the datasheet of the Phy chip you will be using. It should have a lot of information for using separate magnetics.
Yes, important to note there are different magnetics for 10/100 and 1G. AFAIK, the latter is backwards-compatible, but the trick is it requires an H-bridge driver, not push-pull. Traditional 10/100 output stage is just a pair of common source transistors, push-pull around a CT primary -- the CT is mandatory. Termination resistors are also placed there. The H-bridge is usually designed with internal termination, so all you really need is just the isolation. I've seen a few 10/100 PHYs of this design, used just for power savings I think.
Usually they'll still use a CT transformer, but just recommend the CT be bypassed with a cap, no VDDA or anything.
So yeah, check the datasheet, make sure it's happy, use the appropriate kind and all that.
Tim