You need to do more granular measurements. Move the GPIO toggle around the stack and see what takes this long.
But also, why does it need route discovery? Normally device will have the route saved from the first wake up. So it is likely that your receiving device does not send an ACK, which would make sending device wait for it.