Another possibility is to route interesting signals through to the digital output connector on the front
Now there's an awesome idea I didn't consider! 
For me, it would be a good exercise, since I've little flight time on FPGAs. I'm also more familiar with the Altera tool chain; I find it easier to work with and faster than Xilinx ISE too.
Fortunately, as you say, there's little use to decode the SOC<->FPGA SPI comms. It is a bit of a curiosity given the repetitive chatter when running a 'simple' /dso/app/test_cmd. It overflows the buffer on my analyzer due to all the sck transitions.
U27 is a PITA. I've overlaid the top/bot PCB images and at least 4 of the Y inputs end in blind vias. Any idea if the chip selects source from the FPGA? Also, what's with all the inductors on the ADC?
Also, Have you verified the front end serial shift registers that operate the relays?
Check the UCF file! I probed all of those pins already.
# U24/U26 front end relay control
NET "ferc_dat" LOC = "B3" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 0, IO_L4P_0, U24.14
NET "ferc_clk" LOC = "B2" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 0, IO_L2P_0, U24.11, U25.11
NET "ferc_lat" LOC = "A2" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 0, IO_L2N_0, U24.12, U25.11
# U27 Analog Mux
NET "mux_s<0>" LOC = "T12" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 2, IO_L19P_2, U27.11
NET "mux_s<1>" LOC = "V13" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 2, IO_L14N_D12_2, U27.10
NET "mux_s<2>" LOC = "U13" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 2, IO_L14P_D11_2, U27.9
# U30 ADC
NET "adc_sclk" LOC = "V16" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 2, IO_L2N_CMPMOSI_2, U30.15
NET "adc_sdo" LOC = "U11" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 2, IO_L23P_2, U30.13
NET "adc_sdi" LOC = "U15" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 2, IO_L5P_2, U30.12
NET "adc_cs" LOC = "V15" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 2, IO_L5N_2, U30.11
NET "adc_eoc" LOC = "V14" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 2, IO_L12N_D2_MISO3_2, U30.10
NET "adc_convst" LOC = "U16" | IOSTANDARD=LVCMOS33 | SLEW=SLOW | DRIVE=8; # Bank = 2, IO_L2P_CMPCLK_2, U30.9
All of the select pins on U27 are routed to the FPGA via ferrite beads (L46-48). The ADC connections to the FPGA are also routed via ferrites (L49-54). The front end control shift registers are also connected via ferrites (L40-42). I believe they put the ferrites in there to prevent noise from the FPGA coupling into the analog components over the signal lines. Not a bad idea, I suppose.
On U27 there seem to be two analog inputs that are connected to caps on the other side of the board. I have no idea if there is a trace on the inner layer that runs off somewhere for those inputs. I'm hoping that there is a way to route the arb outputs back around to the ADC for self calibration or self test purposes. It's possible they're connected to one of the front end relays, perhaps the unknown ones X4 and X5. It's possible that's how Hantek implemented some of the modulation - send it out the door and back in again to save on the routing in the FPGA. I did not really play around with the modulation much before tearing it apart, so I'm not really sure what it's capable of, at least in terms of one channel modulating the other one. A bit idiotic if they did it that way, but whatever.