Hi,
In my opinion, the DC/DC is simply running outside of his frequency/duty clycle safe area...
If you want to output 5V from 12V, the duty cycle will be 5/12=42%.
If you want to output 5V from 24V, the duty cycle will be 5/24=21%.
Given the inductor value, 21% could be outside the safe area for the switching mosfet (Ton too short).
To fix this problem, you should try to increase the inductor value (keeping the same amps ratings), or (better) try to increase the timing capacitor (internal osc cap).