If you're going to get a GPIB box of some sort, I highly recommend a GPIB to LAN box such as the old Agilent E2050. THey can be had on ebay (from time to time) for about $100. They will expose a VXI-11 interface on the LAN side that will work with pretty much anything capable of talking to LAN based instruments. It's also very convenient just to connect the box to your home network so you can just connect to it over wifi.
I also have a set of Python libraries (Python IVI) that I have developed over the past couple of years that can communicate with serial, VXI-11, GPIB, and USB instruments, it can leverage PyVISA as well if it is installed, and it's been verified to work in Windows, Linux, and Mac OS. The serial port support is pretty good; it just wraps pySerial. I have not had any issues with delays, but I have not used the serial interface extensively.
GPIB interfaces are supposed to not affect the bus when the instrument is off, and I have not had any issues with this so far (with mostly HP/Agilent gear and a bit of Tektronix and Colby thrown in for good measure). One potential issue I have heard of, though, is the drive strength of the GPIB controller. The cheap ones don't always use the proper drive chips, and as a result they don't always drive long calbes/lots of endpoints very well.