It might be useful to review the OSI seven-layer interconnection model:
https://en.wikipedia.org/wiki/OSI_modelBecause you are using Cat-5 UTP and 10BASE-T, you are using layer 1, the PHYSICAL layer.
That is not "Ethernet" at the LOGICAL layers (data-link or network or transport, etc.)
You can take advantage of the cable, connectors, and transformers pre-engineered for Ethernet.
That does NOT mean that you have to send "Ethernet-like" packets, datagrams, etc.
It only means that you are exploiting the already-available sending, transmission and receiving engineering designs.
And you are able to send a bit-stream into the transmitter driver, and get the bit-stream back out the other end.
That bit-stream can be the continuous ADAT data-stream. It does not have to be the "Ethernet" protocol of addressed and routed packets.
You seem to be making this a lot harder than it really is.
Yes, I agree that learning how to transformer-couple a signal is a good thing to know.
And winding your own coils and even making your own capacitors and resistors was instructive also.
But here in 2016 there are more useful things to learn and know beyond pulse-transformer engineering.
Pulse-transformer engineering is rather complex and arcane. In English we have an idiom: "bigger fish to fry".
Which means that you have more important details of your project to work out beyond 10BASE-T physical transport.
I am just afraid that you will get bogged down in working out optimum transmit and receive coupling for your Cat5, and that will delay working on the heart of the project. That is why I suggested using pre-engineered "Ethernet" transformer solutions which are a very mature and very inexpensive solution.