Besides setting the low frequency cutoff, a low value of capacitance will introduce distortion if a ceramic capacitor with high voltage coefficient of capacitance is used or an electrolytic capacitor. This is because when the voltage across these capacitors changes significantly, then the capacitance does also. So to minimize distortion if these capacitor types are used, the capacitance must be much larger than required for the low frequency cutoff.
Some electrolytic capacitors can be noisy. Poorly made metalized paper or film capacitors can be noisy because of short clearing. Sometimes circuits which must have guarantied low noise use metal foil capacitors. But in general noise is not something to worry about in capacitors.