Hence why i am waiting for IOMMU pass trough to be more mature in VMs so that i can have a Windows VM with full GPU acceleration and all that. Then i can justify going to Linux as i always have an option to spend all my time in the VM. If things don't work out, nuke the Linux boot partition and just boot the windows drive directly outside a VM.
IOMMU is mature and works fine, but you have to use new HW that supports VT-x and, in particular, VT-d "directed IO" (or AMD equivalent features). You can't expect the kernel or user space software to fix something that requires a HW feature... It works fine on at least 11th gen Intel Tiger Lake and Rocket Lake systems (what I currently own). If you are running something older, I couldn't say for certain, but you can look up your CPU on Intel ARC and see what features it supports (If you have AMD, they might have an equivalent feature table but I haven't used it). If your CPU has VT-d, you are good to go. You can only pass over PCIe devices. Otherwise you will use paravirtualized devices for the guest HW. I pass over an entire NVMe drive and my second GPU and sometimes the Thunderbolt 4 controller and have also passed over the SATA controller at one point. USB passthrough, if I'm not mistaken, is paravirtualized. To get the best performance, I'd just pass the PCIe connected USB host controller (if you have more than one) and get native performance without all the indirection needed for host to guest USB operation. This is especially important if you plan on updating firmware on a USB device, as you need the controller to be present on the guest for it to succeed.
You also need to enable all the kernel features at boot time with "intel_iommu=on iommu=pt"
Linux is great, but is never going to be a turn-key option for a random laptop or collection of off-the-shelf parts for a custom PC. You will likely need to adjust your kernel boot time arguments to work around issues with your specific HW. And don't use Nvidia if you want to have a fully functioning system. Seriously. Until NOVA makes it into the kernel and the closed and open user space graphics stacks use it, using Nvidia is not going to be a good time. Even then, only Turing (16xx / 2xxx series and up) are going to be supported. If you have anything older its the out of kernel Nvidia module and closed source user space stack, which breaks if you update your kernel and doesn't like Wayland at all. So you'll be stuck on x11 and that is being depreciated by seemingly everyone at this point. Now, if you have two GPUs (e.g., Intel/Nvidia), passing over the Nvidia card is a great option.
And Looking Glass (which is a pain to setup correctly) is incredibly performant for local use of the VM when doing work with CAD, image editing (Adobe) or gaming.