Thank you for the suggestions. The ESP32 tool seems good even for normal use because of Wi-Fi support.
Also, apparently there is an official tool called Hardware Server for Windows by Xilinx. It is around 500 MB and by launching the program on local machine you can connect to the hardware server that is launched on local machine from remote desktop. I don't even need to copy the generated bitstream files or anything. The only difference is Vivado Hardware Manager shows the hostname instead of localhost on the Hardware Manager page.