dsub 9 is not particularly small by modern standards, but it is cheap, rugged, has lock screws, enough pins, and is easily available in PCB mount. Modular telephone jacks (RJ11, RJ45) are less rugged, generally only support unshielded cables, and their latching is fairly weak, but they are super cheap.
M12 circular connectors are smallish, a lot more rugged and not terribly expensive. The M8 connectors i think are only available in up to 4 pin.
LEMO and Fischer make locking circular connectors that are relatively small and very rugged, but be prepared to spend an entire day trying to figure out what you want and making sure that the parts you order will actually mate. Then you will pay upwards of $30 for a mated pair for the privilege of waiting 12 weeks for them to be delivered.