(I was waiting for others to reply, but they didn't, so I'll give my opinions.... Perhaps this would have been better to post in the RF forum. My below response is quite hand-wavy, probably not quite right for the "metrology" forum, but may help you ask better questions )
Looking at the manual, it seems to describe only a transmission measurement, but not a reflection measurement. I don't think that with the single internal splitter, you would be able to measure the reflection. Or... you're adding an external power splitter?
The E5100A can measure the gain/phase between any of its four inputs (R,A,B,C), due to its use of a DSP to do the phase/gain acquisition (instead of an I/Q demodulator like "analog" VNAs). Based on its block diagram, it downconverts the input signal to 12.5 kHz, and then bandpass filters it. This filters out most interference (so a "pure" sine wave is not required). It seems that there is no image rejection filter before the mixer, so it could be confused by signals the reference frequency +/- 12.5 kHz (unless they do have image rejection and just didn't show it on the diagram). The ADC's frontend does have an anti-aliasing filter, so the DSP can pretty accurately discriminate input frequencies.
Most power splitters (such as Wilkinson) have reasonable directivity (~20 dB), so the reflected signal doesn't influence the "R" port as much as you may expect (hopefully your DUT is already well matched. With a well matched DUT, the reflection IS small.). Yes, they will be input to the RF source, which isn't so good, but the splitters have enough loss that I don't think the amplifier would be damaged. VNAs only output low source powers, anyway (<10 dBm).
Yes, filters must either reflect or absorb the undesired frequencies. With the 3 dB splitters, the signal's power will be halfed during transmission and reflection, so the power incident into the PA will be no less than 6 dB lower than its output power. Most PA are perfectly stable with this. The phase is not "random". The phase of the reflected signal is a function of the filter and is stable... it doesn't change.
In the case that you are using both the internal and an external splitter, you're using the external splitter as a 3 dB directional coupler. It still will have decent directivity (> ~20 dB), so things are not as bad as you think. In addition, the VNA's calibration (with a SOLT calibration), is able to determine the and calibrate-out the directivity of your splitters/couplers. I'm bad at the math (and not looking it up at the moment), but having both the open and the short is helpful to figure out the directivity/coupling factors. The correction model you're looking for is "enhanced-response", which is a one-port calibration, plus response (thru) cal. This would use four standards (short, open, load, and thru), and I think is an 8-term error model.
Finally, to answer your question about reflections, we usually like |S11| to be > 10 dB, which is 10% of the power being reflected. When you look at precision things, the reflection is usually specified as VSWR, like 1.9:1 (which is around 10 dB). Good connectors will have VSWR ~=1.05:1, which is 0.06% power reflected.