If you mean that the joystick shows an offset to one side (e.g. you character in a game keeps moving or camera turning) even when you are not pushing it, that's much more likely to be a calibration issue (i.e. software problem) than anything to do with hardware.
Joystick "drift" just means that the joystick is not centered exactly (it never is - manufacturing tolerances, etc.), so you need to calibrate those offsets out and possibly add a "dead zone" (threshold) around the middle position. I am pretty sure XBox has a controller calibration utility, try to use that.