The core contributes core losses, but how much depends on how many turns you've put on it. The equivalent circuit is a resistance in parallel with your source. Presumably, you would want the core loss equivalent to be several times larger than your source, so that its contribution both in noise and attenuation (and inductive reactance, which will be dominant) is small.
A >100kΩ impedance is not all that reasonable for a ferrite transformer, certainly not over any kind of bandwidth. If you are making a narrowband / tuned antenna, in a low (SW or below) band, it may be possible.
A much better solution is to simply use a lower impedance amp in the first place, transforming as needed; this can be done wideband. The high impedance amplifier is simply a poor choice here.
Tim