Plastic for dust, unless it requires special termination or otherwise susceptible to outside interference--not much in my case. Some metal caps on rarely used ports, just because I have them.
I find the the hard plastic top hats easier to use and longer wearing than the rubbery caps, particularly with SMA.