In my experience, analog servos do not make noise when idle, unless torque is applied. Then the motor will need to be active to keep the position, and due to the analog circuits which control the motor, you will hear a buzz at 50 Hz (assuming a 20 ms or 50 Hz servo pulse repeat rate). If the servo is twitchy, then you have either: dirty/noisy power, or a noisy signal. The signal may be noisy due to external electrical/RF noise, or poor quality (not stable) generation of the signal itself.
I just looked at servo.cpp from whatever Arduino version I have here. It uses a single timer interrupt to trigger a digitalWrite() for the servo pulse high/low transitions. This is susceptible to jitter. Basically if the MCU is busy, especially inside another interrupt handler, then this interrupt handler might need to wait, and that can and will result in the servo pulse being +/- at least several microseconds over time.
I built a project for an R/C car which generated two servo outputs using the ATMega328P Timer 1 output compare A and B to generate the two signals, with perfect jitter-free timing. I fiddled with the timer registers directly rather than using a library.