You will have potentially live voltage with minimal to no isolation exposed on the USB connector. It is just a bad idea all around.
You can use switching power supply, and small transformers are pretty cheap. You can probably even find complete sub-assembly at a decent price.
Your other option would be to power all low-voltage part from USB +5V, if device is constantly plugged in.
Or power FTDI from USB and use optocouplers to decouple it from the high-voltage part. Although this one also depends on what else on that board is accessible by the user. You don't want to have mains on encoder knobs either.