The point of this setup is also run time/expandability, so a regular UPS does not quite cut it, the UPS I have is actually an inverter-charger hooked up to marine batteries, though it's still like a UPS where it takes 16ms or so to switch over. 99% of the time it works fine, but clearly I must have got unlucky and hit the 1% that it didn't. The server PSUs are high quality hot swap units. I really doubt they are bad, though since they are redundant and hot swap I could easily take one out to inspect then inspect the other without bringing the server down again. I flipped the breaker on and off a bunch of times and it never happened again so this was a weird fluke. It took a bit of effort to throw the main and there was a bit of arcing, so I wonder if that was an issue, I've seen situations where there is a brownout and the UPS does not kick on fast enough. Common with laserjet printers plugged into the same circuit as the UPS (obviously not plugged IN the UPS, just along side it)
I never thought of looking at solar stores, I actually do want to look at adding solar to the house at some point, so I could possibly look at a charge controller that also has commercial power capability to keep the batteries charged up.
If I was to actually go the custom route, how realistic would it be to design a PSU that is like 90% efficient and will last long, for a newbie? Could that be achieved simply using a buck converter topology and high quality components? I'd probably build one design that's like 500w then just parallel them. I've been wanting to play around with making a bench PSU anyway, I may start with that to get a feel for it and go from there.