It might be the oscillator, but it might also be other things, like crosstalk from other high speed signals in to the clock trace, it could be a noisy power supply for the oscillator or the PLL, or it could even be wonky PLL settings. Fancy complex PLLs can be unstable in certain configurations.
Jitter can be easily visible on a digital scope by triggering on the clock and then turning the horizontal offset way out so you are looking about 1ms after your tigger point. For a proper crystal oscillator the waveform should not look fuzzy or wander left or right any. This method is sensitive enough to see the waveform drift when you heat up the crystal and see its tiny temp co.