The isolation transformer is to put between the RCD and the LISN input, which will stop it tripping, as it stops the RCD from seeing the leakage current. This is where isolation transformers normally are used incorrectly (bypassing the RCD decreases safety) but in this case there is no choice.
The isolation transformer is NOT there to let you float any test equipment or for any safety purpose, and the earth connections must be maintained throughout (indeed they are of critical importance for safety and measurement integrity).
The equipment you are testing should be connected directly to the LISN mains output as specified in the EMC testing standards, with no RCD etc. You shouldn't really be working on the equipment under test while connected to the LISN other than for testing normal operation - certainly don't do anything where you'd be wanting the protection of an RCD (this is a good rule in general, relying on one tripping to save yourself isn't smart - they're there for a last line of defence backup only!).
Sorry to go on about it, but this stuff is super important for safety. A LISN is an unusual piece of test gear in that it creates serious safety issues if used incorrectly. At the very least, as well as following the manual instructions for setup, all protective earth connections on the LISN and the device to be tested (if applicable) should be tested for a good solid connection to mains protective earth before powering anything up. The RCD tripping is actually a good sign, it's doing what it should and also shows that the earth connection is there.