Definitely not the right way to do it.

As you say, for an S11 measurement, the VNA is acting as the signal source in this scenario. The VNA should really be the only active signal being sent. If you want to look at the output of a transmitter, you would be better off using a power meter or a spectrum analyzer or something. for safety, you might do as you say and include an attenuator in line with your measurement, but you will want to calibrate it out of course.
First off, under NO circumstances should you ever exceed the max input power of the VNA. Just to check, if you leave the VNA port open circuit and set it to measure S11 with correction turned off, you should see a squiggly line at around 0 dB. If the S11 line is closer to -10 or -20 dB, or has big nulls or other weirdness, it is possible that the input is blown. you could also try running the operators checks. you'll have to look up how to do them for your particular model of VNA.
In order to measure the S11, you should be measuring everything AFTER the transmitter, such as any cables, filters, and whatever the transmitter is going to (an antenna, some other part of the system, etc.)
Depending on the type of circuit, it might be possible to get some reasonable S11 data by trying to measure the transmitter with the power off. If there are any connectors, vias, or traces on the board that you suspect are creating high VSWR, you might be able to isolate the issue using TDR gating if your VNA has it.