The function uses the _XTAL_FREQ to calculate it's timings etc, but unless the physical clock is set correctly then it won't work properly.
If you set the frequency to 16MHz but tell the delay function you have a 8MHz clock then things are going to go wrong, don't forget that the delay functions just waste cycles, personally I use a hardware timer with flags.
With the internal clock you have to make sure the clock frequency bit is set correctly in the register, otherwise the instructions will not execute at the correct speed, they will execute slower or faster, depending on which way the clock was wrongly set.
If you have a RTC clock at 32HKz, the software is set to tick at a specific rate based on the knowledge that the input clock is 32KHz, if you suddenly change this clock for a 1MHz crystal for example, then the RTC is going to be counting at about 30 x the original speed.