I tried a couple years ago, and gave up.
Problem is, as far as I know, GCC never included the PIC32 target in the mainline source code, whereas a lot of other targets are part of it. There's probably a good reason. (Funnily enough, I managed to build GCC for the LM32 target (LatticeMico32, Lattice 32-bit soft core) without any problem.)
So anyway, your chances of getting to build XC32 with any other version of GCC than the one Microchip used is close to none (and even with the right version, it's a huge pain to build). And even if it built, the probability of compiler bugs would be very high, so I wouldn't bother.