That helps a lot. It is a brand new, very efficient motor, so going to a BLDC setup isn't going to help all that much. It's interesting that they list 50Hz specs, although there doesn't seem to be any real design involved--almost any 60Hz 3-PH motor would behave similarly. Note the 1.00 service factor at 50Hz.
Does this motor come attached to the pump as a unit or are they two separate items that you have combined? If the pump/motor is sold as a unit, does it have specs for flow and head pressure?
So I get 3288kVA and 2926W at full service factor load at 60Hz, not counting any VFD losses or tolerances in specs for the motor or the inverter--both of which could be rated optimistically. Better than my earlier estimates, but still either cutting it close or just too much, depending on the loading. The 0.6 PF of the VFD may not be helpful, but you're still at or near the limits. The fact that reducing the VFD frequency to 54Hz extended the run time seems to back that up.
One other thought--you mentioned that the pump ran for 15 minutes at 54Hz. If the input to the inverter is, say 3.2kW, that is a roughly 66A load on the batteries, more than C/2. Have you checked to see how low the battery side of the inverter is going? That 120Ah is likely the C/20 rate, at more than C/2 you are pushing the limits there too.