The cap on the LM199/LM299/LM399, which is for thermal insulation, has no rigid attachment to the package, so it is normal that this cap can move around a fair bit. It just surrounds the package. If the reference itself would actually be loose, the meter wouldn't be able to give any stable reading and would probably give a power on self-test. So if the readings are even slightly sensible, you can be sure the reference is solidly connected to the socket.
Both pictures look normal to me.