After letting everything sit overnight, here is the latest info.
* The (big) A6304XL probe repeatedly degaussed without issues this morning.
* It also degausses properly on the second AM503B that we have. Behavior is identical.
* Both AM503B's also operate properly with the smaller A6302 probes.
* The A6304XL takes at least 3X the time to degauss compared to the A6302's. True on both amplifiers. You can see the waveforms on the scope and hear the relays clicking, just as with the A6302's, but each phase takes a bit longer. The real delay comes from how the process seems to stall on certain steps with the big probe. It sits there for so long, with the red light still on, that even when I know it takes longer it feels like it's going to fail. But patience pays off, and eventually it advances to the next step in the sequence. This happens at several points along the way and the cumulative time is quite lengthy.
My conclusion is that the much larger ferrite mass in the A6304XL requires more time to degauss and balance/zero. The AM503B's must be watching feedback signals during the process and on the big probe it just takes longer for those values to get where the amplifier deems necessary before proceeding to the next step. As for the initial failures, it may have been sitting unused for so long that the core took on a large bias that required multiple sequences to finally zero out. For now, I'm just thankful that it all appears to be working properly.
TLDR: Be patient and retry the degaussing sequence, several times if necessary.
Hope this helps someone in the future!