The difference between an 11 GHz and 18 GHz connector of the same type is mostly precision manufacturing. A lower spec connector, one that has been abused, or not torqued correctly will have worse insertion loss and worse return loss at high frequency because it will not present a constant impedance and may have more material loss.
If you go higher in frequency the other factor is multi-mode behavior when the frequency exceeds the waveguide cutoff for the ground shield. This is what happens if you try to use an N connector at 40 GHz, and why 40 GHz connectors have to be so small. This causes resonances and beating between the modes. Any bend in the cable will cause mixing of the modes so the performance is not just bad, it is unstable as slight changes can cause a dip to turn into a peak or vice versa.
When talking about coaxial attenuators and filters, multi-mode behavior also degrades performance significantly since the waveguide modes typically bypass the attenuator or filter component unless you take special precautions. For instance, a segment of coaxial transmission line with the dielectric filled with eccosorb can be made that will heavily attenuate from ~20 GHz to visible light (literally!) including waveguide modes. These filters can't really be made with precise or sharp cutoffs but you can cascade them with a conventional lowpass or bandpass filter and make a filter that has no higher order transmission.