After almost a day (!) debugging I got it working.
Turns out the FT232R dongle I was using was spitting out extra bytes - specifically 0xFC - often at the start of a packet sent. I'm not sure if this is due to how Microchips Host app ("UnifiedHost-1.19.2.jar") communicates with it, or if its an FTDI driver issue - note this one uses a chip I bought from Farnell back in 2009 so I very much doubt its a fake that FTDI drivers have screwed with.
I used a CH340 dongle to snoop on the data sent between host (FT232R) and micro (PIC16F1704) and noticed the extra bytes. Always 0xFC.
So, swapping the dongles so the "good" CH340 was used for the bootlaader, got the two talking, but the host would spam the same packet - trying to erase flash but sending '0' for the number of bytes to flash, so the MCU just looped forver. Out of curiosity I switched to a different host version: UnifiedHost-1.19.0.jar
I couldn't find any release notes about versions on Microchips website, so I have no idea of the differences, but this worked, and pretty much immedaitely flashed a blinky app to the MCU.
Finally (what a saga), there was a bug in the generated bootlader code which didn't read the config words properly, this had to be re-written so it had the correct starting address, and read the high byte as well as the low byte (it just read the memory linearly, addressing words, but only reading out single bytes rather than reading the entire word and splitting it to two bytes).
All this really doesn't inspire confidence in Microchips bootloader - there is one user guide pdf, but thats pretty much it, no other information on their website, and it seems very few people bother using it - at least not for PIC16. It's got quite a small footprint, so thats nice, I think I'll just write a new host app (C#, .NET, windows only) so that it actually remembers all the address offset settings for different micro's, so I could jsut load a *hex, select the micro to use, and hit send.