As Bud says, start with an external linear power supply first to eliminate the USB as the source. The 5V USB is definitely going to be noisy, and might have high ripple and switching transients on it. Use a scope to see if this is true for you. Using a linear PSU to power your ADC and reference, you will want to first get the system working and get your 5 noise-free digits. Then go back to the USB power and see what happens, and what you need to do to clean that up.
Also check your decoupling capacitors on AVDD and DVDD. You will need good decoupling, 10uF and 0.1uF at least as a rule of thumb to start with. Further analysis and determining the source of any noise might lead to a few smaller caps needed still.
Although I don't think the reference is the source of the noise, at 24 bits you might want to low-pass filter the output of your reference before going into the reference input pin of the ADC. Ideally a buffered filter is required, since the ADC reference input will draw current when taking a measurement, and even 250uA drawn across a 100 ohm resistor can result in 1% error at 2.5V. A simple RC filter can work, but don't make the R too high, or it will cause said voltage drop and error. Perhaps try 10-ohm, 470uF for a 33Hz filter, but you probably need the 3dB point an order of magnitude lower still, around 3 Hz. These RC filters have large time constants, so you might need to wait a minute or two for the reference to settle. Also, you can enable the REF buffer in the ADC to help reduce the current drawn from the reference (which helps reduce the voltage drop due to your RC filter, which might allow you to use a larger R and smaller C again). Ideally, in practice you will want to use an active low-pass filter so you can use smaller C's and get a lower breakpoint.
Filtering your reference may not be necessary in your application, so try that as an added step if you're still seeing noise but you've eliminated everything else already.