Generally, they're all brought together to a common ground anyway, but...
There are often ferrite beads used on the RGB pairs. Sometimes it's only one for all three. In which case, common grounds won't matter (in principle, at least).
Often, they are done individually, though, and shorting the grounds together elsewhere will partially foil the intent of those ferrite beads.
So, better to leave them separate. Also, keep the shield tight in, as close as possible and all that. I don't know what the backshell looks like, but I'm guessing it's not crimped coax joints, so... anywhere you have to separate shield from core, and wire them to pins, stuff like that.
It really only matters with noisy environments and long runs, but if you want best quality or operation, it's always better to do it well.
Tim