Maybe I'm wrong, but one signal comes from the PC side, and the other from the connected device. The DTR reports that data has been received and needs to be read, it comes to the PC from the device side, and the RTS signal is used to turn on the transmitter and is fed from the PC side.
Not correct - both DTR & RTS are sent by the host computer. What needs to be remembered is the origin of the system - think two computers connected to one another via an analogue telephone circuit with a modem or dataset at each end.
DTR or Data Terminal Ready is asserted by the computer host to inquire if the communication device (the modem or dataset) is ready to receive data for onward transmission, and the dataset would then respond by asserting DSR or Data Set Ready. Once the DTR/DSR handshake has been completed, the computer host would then assert RTS or Request To Send and wait for the CTS or Clear To Send to be asserted before actually transmitting data.
Generally no DSR meant the dataset was not powered on or present, and no CTS meant the dataset was not connected to it's partner at the other end of the link
Please note that we have only looked at one half on the communications - one computer to it's associated modem or dataset - we have also not touched on how the first dataset makes a connection to the second dataset - that is not really relevant to the discussion, in the old days a human picked up the telephone & dialed the number to establish the connection and then put the phone handset down in an acoustic coupler.
The other half of the communication can be another computer attached to a second modem, a printer attached to a modem, a "dumb terminal" (meaning a display with keyboard but no processing power), again attached to a modem - we are talking pre 1970 technology here - no ethernet, no tcp/ip, no internet - none of the stuff we now take for granted.
Where things get confusing is when the telephone line & modems are removed from the "picture" and the various "hardware handshakes" have to be dealt with - the easiest way is to disable the hardware handshake altogether (it's not really required with a direct cable connection because you don't have to worry about dropped calls), you can also jumper RTS to CTS and/or DTR to DSR as required by the specific situation.
Determining what was actually required would be done on a case by case basis by inserting a breakout box (see below), observing the LEDs and then inserting jumpers as required - once the desired configuration was known the interconnecting cable could/would be hand built.
This is a dying art (it may be dead already), I still have a break out box on the shelf, but haven't used it in maybe twenty years.

By the way, none of this is really relevant to the original poster's dilemma - the PC has to assert DTR and/or RTS - so whatever software he's using has to be told how to do it.
the software only brings the DTR High to trigger a transistor to ground a pin to start transmitting.
the RTS line is set high to trigger a transistor to ground a pin to shift the carrier 170Hz and then pulsed to send the data.
This sounds quite unusual to me - as mentioned above DTR & RTS are control signals - I can see DTR being used to enable the transmitter, but pulsing RTS to actually transmit the data is contrary to normal RS232 usage. What I would have expected is a UART connected to the RD & TD (receive data & transmit data) lines, and that UART then passing the data to some sort of microcontroller that handled the keying.
Having said that, I will be up front with you and say I have ZERO experience with RTTY, I'm just an old head that grew up in an era when RS232 serial communications were the way it was done.