What I think is happening is simply the OP is hitting the speed limit given by the combination of motor back EMF, and maximum drive voltage.
if that is the case, usually two different routes are viable:
1) usually with native three phase inverters flux weakening is used, where a part of the driving current is used to counteract (and thus lower) the rotor magnetic field, this allows to widen the operating speed range, of course thermal limits still apply and since the part of the current employed to lower the back emf can't generate torque, the higher the speed in the FW region the lower the torque. However I'm not sure that with your setup (driving just 2 phases and not all three) its doable
2) use a higher voltage to begin with (in this case higher transformer turn ratio or higher starting voltage)
in any case I strongly doubt the problem is a phase imbalance, also be aware of the mechanical speed limits of bearings, couplers, the rotor itself etc, as you are already 6 times higher than rated spead, and going higher the machine could suffer a rapid unsheduled disassembly