I would avoid socketing the switch to the motherboard. Not only does it lock you into an exact brand and model of switch (so a switch failure would mean a motherboard replacement if you can no longer get the exact same switch, and with cheapie switches it's not a question of if the switch will fail, but when), but it will take up room on the motherboard that would better be used for the RPi's.
The neatest solution would be the SPI Ethernet, but this would be very expensive, as it would require duplicating the SPI Ethernet circuit for each of the 32-64 RPi's. Not to mention how slow it would be compared to the 100Mbit Ethernet, let alone to the 1Gbit on the OPi 2.
However, it looks like you'll have plenty of room under the motherboard, so I would route out a small slot next to each board and run the Ethernet cable through the motherboard, up behind it and over to the Ethernet switches above. You'll need jam in a whole bunch of those little switches up top, but there should be enough room if you rip out the optical drive and everything else. For 64 RPi's you'll need 10 of the 8 port switches - 7 RPi's + 1 uplink per-switch. 16, 24, or even 48 port switches would be better, but I guess it depends what you can get for a reasonable price, and the total number of RPi's you want. If you're running OPi 2's or other gigabit boards, you probably don't want to run all the switches to an aggregation switch, with just the one uplink to feed the lot, since that would effectively give you a maximum of 64:1 contention ratio. Even if you're using 100Mbit boards like the RPi 2/3 or the OPi 1, you'd still have 6.4:1 contention ratio. With 8 port switches, you'd have 10 uplinks, so I'd just bring all the uplinks to the back panel. I would cut out the connector panel, either to the right size to snap in those Keystone Cat6 Ethernet couplers, or mount sockets along the top of the motherboard to pass the Ethernet signals through to sockets on the back panel. You could even 3D print a custom connector panel to get the exact port layout you want.
But then again, I am a network engineer rather than an electronics engineer, so obviously I'm a bit biased when it comes to network connectivity!
As for power,
here's the G5 "service manual". The power supply pinouts are on pages 179-180. The cheapest solution would be a Wun Hung Lo buck converter off ebay (massively over-rated to allow for their huge power rating fudge factor). Building one into the motherboard wouldn't cost much more though. The toughest part would be finding somewhere out of the way of all the RPi's to mount the heatsink for the mosfet and power diode.