Yes, the field that's not within the wire. Suppose you have a straight, infinite wire with radius R1; the magnetic field which contributes to its inductance extends from R1 to infinity (and a little bit less than R1 due to skin effect, but suppose we count that as zero for now). Now consider a thinner wire of radius R2 < R1. At a given current, the magnetic field from R1 to infinity is exactly the same -- Ampere's law states this. The field only from R2 to R1 -- which by definition is not part of infinity -- is exactly the field which causes this thinner wire to have a higher inductance (or impedance).
Now, it's worth noting that wires don't go for infinity anyway; infinity is a rather poor conductor. So suppose you have a pair of wires. Now you have the capacitance between them, as well as the inductance between them; and this defines an impedance. A pair of wires very close together has low L high C; since Z = sqrt(L/C) the impedance is low. The stray field is also low because what little field contributes to the inductance is mostly between the wires (for a separation D, you'll find very little beyond 3*D or so in any direction; though it is never zero, so does extend to infinity). In this sense, "higher impedance [transmission line] will broadcast more noise..".
But that's still not accurate, because there are many configurations of transmission line. A coax cable is self-shielding and radiates nothing, regardless of impedance. A parallel-plate or ladder line will radiate as a dipole. Twisted pair radiates as a dipole if the twist is unlucky, but that occurs for few frequencies, so it can be assumed on the average to cancel out; it's not self shielding like coax, but it does a pretty good job for the most part.
So it would be more accurate still to say: "higher impedance, untwisted, parallel conductor type transmission lines will broadcast more noise..."
By the way, this assumes common mode is zero. Any transmission line, driven common mode, reduces to the case of single-conductor-in-space, so it is not necessary to consider combinations. Common mode noise is often the primary concern in switching circuits anyway, since that's the part that causes conducted emissions.
Tim