I've recently done a few toy projects with PIC32, and although not perfect, it's also not that bad IMO.
XC32 works up to O2, which is good enough for most applications. I believe XC32 was updated to GCC 6.x branch a little while ago, so it is atleast possible to use some of the newer language features in C++14, if that is of interest to you.
But unless you're doing something fancy in C++ with complicated constructions for virtual method devirtualization, you'll unlikely observe massive speed/size improvements going beyond O2. The compiler will (under the hood) compile your code twice for -Os as well, and advertize the binary size in that case (so you buy a license

)... It's slightly annoying and makes compilation a bit slower, but not unusable by any means.
Oh and the specs modification still works, but I only tried it on my office machine, as I can't be bothered to put it on my lab machine. Like, honestly, for my projects I didn't have code size or speed problems (even though I know some code will alot run faster on O3), so I don't see why bother.
As in terms of actual silicon, the MX series are not bad chips. Their Coremark/MHz score gets close to 3-3.5/MHz, where a Cortex-m3/m4 will score similar. The documentation and programming of peripherals is rather straight forward.
There are perhaps 1 or 2 things that I find lacking somewhat. First is the
quantity of peripherals. It's easy to get used to having 3.6MSPS 12-bit ADCs, 6 SPIs and 8 UARTs on a 100-pin STM32F4 chip. But if you can fit the I/O layout, it's doesn't have to be a problem.
The energy efficiency of the MX chips, to todays standards, is not great.
Personally the reason for me that stopped using PICs a while ago was their IDE. I don't like Netbeans and the glitchyness, how slow it is, while using plenty of resources on a fast machine (12-core AMD 3900X). For my ARM projects I've switched over to CLion, so I expanded my CMake scripts to support the XC32 compiler. However, I have not been able to get debugging to work so far from CLion. I would need a XC32 gdb executable (probably), but more importantly a GDB <> PICKIT bridge. There were some opensource alternatives for that, but it didn't work out yet for me. I may try openocd + JLink on JTAG pins one day.. but unfortunately I've only broken out the ICSP pins on my boards.
However, there is the mdb command line simulator/debugger in MPLAB, that can be used independently from the IDE. It's not interactive, but it can aid a little bit in debugging. I've used that to upload debug code, run the program till a fault occurs, and then halt the program to inspect the PC (which using a custom assert() can hint which one failed). Also printf() debugging. Both methods are rather primitive.. but also sufficient if you test application level code on PC first, and only need to debug hardware related stuff on the device.