Hmm I'm not sure why it's floating in that schematic. In the layout it is connected to ground on the digital side, but not connected to analog ground. It used to be but that introduced a loud whining noise.
That 33uF cap with a 47k resistor gives a cutoff frequency of ~0.3Hz, which according to Small Signal Audio Design is too low as it's desirable to remove infrasonic noise caused by the mechanics. I think bumping that up to ~10Hz would be a good idea, I'll try it out tonight.
Everything about coupling to AGND instead of VCOM makes sense. An interesting effect was that if the phono input was coupled to AGND the frequency response would change, as shown in the attached chart. In that case however the rest of the circuit was still coupled to VCOM as shown in the original schematic, which may have something to do with it.
Great info everyone, I'm going to try these things out tonight! I feel like a dummy overlooking what in retrospect seem like straightforward things but what can I say.