For some reason, I always had the impression that VxWorks' direct competitor/counterpart in the open source space was RTEMS. It seems to be used by the same type of sectors too (Robotics, space stuff, etc.)
A lot of Zynq FPGA users judging from the mailing list, plus SPARC.