Diagram A in the manual you attached shows the relationship between the RJ11 pinout and the standard DE-9 PC COM port pinout.
To identify the wires:
The ground wire should have continuity back to ground on the USB connector, and if you plug the cable into a PC and open the port in a terminal program, with no data being sent, one of the remaining wires will be a floating input (RxD) which will measure close to 0V and the other (TxD) will output a voltage: +5v or +3.3V if its a logic level (aka: 'TTL') USB serial adapter, and negative -5V to -25V (but unlikely to be more than -9V) for a true RS-232 levels USB serial adapter. See
http://tldp.org/HOWTO/Serial-HOWTO-19.htmlThe two types of adapter aren't compatible, your inverter expects true RS-232 levels and, as its only got three wires, almost certainly doesn't provide USB Vbus so you cant use a MAX3232 serial level converter chip, so if you measure a positive voltage with the port open, you'll need to get a different adapter (preferably one with a DE-9 plug as they almost invariably use RS-232 levels).