The VCO is in a feedback/PLL loop with PC2out as the comparator (the custom chip must be generating some reference) and R303,R304,R334,C302 as the loop filter. If you have stability issues it is most likely due to the phase comparator and loop filter, unless you have switching noise directly affecting the VCO. Can't comment much further as we don't know what is going on in the custom chip.