Voltage levels are the (maybe unavoidable) cause.
I was writing this simple calculation, in the middle Mike posted his one, which I agree.
For 4 bits, considering that the output level is always + or - V, power dissipated into 3k loads will be Pdiss = V^2/3k * 4. For classical V=12V, 12^2/3k * 4 = 192mW. For V=5V, 5^2/3k * 4 = 33mW.
If it has to be a fully compliant with RS-232, there is no way other than turning off the driver when not needed.