Author Topic: Windows VMs under Linux with Native Performance  (Read 4774 times)

0 Members and 1 Guest are viewing this topic.

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Windows VMs under Linux with Native Performance
« on: August 06, 2022, 09:10:30 pm »
Hi All,

This is a shameless plug for the free open source project I wrote which I use on a daily basis along with thousands of others that allows the use of a Windows VM in Linux that has a VFIO/SR-IOV or GPU pass-through setup.

2 Minute Demo (aka, TL;DR; version  ;D )
https://youtu.be/7XbQOjfnxbU

Before I begin I should explain what this is as it's not a commonly known technology and as such many people do not know that it's even possible. With the advent of cpu extensions like IOMMU and VFIO we are able to isolate PCIe devices and give them to a virtual machine for it's sole use. This technology has been used for years now in the hosting industry where network interface cards support advanced features like SR-IOV/VFIO, allowing them to be split into multiple "virtual devices", or more accurately known as "virtual functions". This is done to give the VM direct access to the hardware so that it can obtain bare metal performance on the hardware.

For a while now there has been a growing community of users that are using this to instead pass a complete GPU to the Virtual Machine, allowing the VM to have bare metal 3D rendering performance. This does require you to have two GPUs in your system, either by installing another one, or if you are using a laptop you may already have two GPUs (iGPU + AMD/NVidia something). This is commonly known as a VFIO Pass-through configuration, or VGA Pass-through configuration. There is one shortcoming of this type of a setup, the GPU has no idea that it's inside a VM and as such still wishes to output to a physical monitor connected to it's output.

This is where my software comes along (Looking Glass - https://looking-glass.io). This is a two part application that makes use of a special virtual device called IVSHMEM, or "Inter-Virtual Shared Memory" to map a block of shared RAM into the VM that can be used to move mass amounts of data with extremely low latency in/out of the VM. The "host" application (not the host system, yes the naming is confusing), runs inside the guest and captures the frame-buffer output of the GPU and feeds it into the shared memory segment.

The client side application that either runs on the host, or even in another VM (yes, we can run VM to VM) then takes the feed in shared RAM and is essentially a high performance RDP client but instead of using a slow network protocol with compression, it's lossless and the latency is outstanding. In-fact in some cases we can get the frames to screen faster then the GPU will output them to the physical monitor as the Linux graphics pipeline is far shorter then Windows.

So you might be now asking what kind of crazy exotic hardware you need for this kind of a setup? That's just it, none. You can do this on any CPU that has a decent core count (6+) and use any secondary GPU you have (provided it plays nice with VFIO, some AMD GPUs have hardware bugs that prevent their use). We have people running this on 5 year old laptops, in use in universities by students and professors for comp-sci projects, and artists that need access to windows only applications such as the Adobe suite, AutoCad, etc.

Personally I have this working on my Intel laptop, a Ryzen 7 1700x desktop, and my current workhorse that was kindly donated for this work, an AMD EPYC Milan system. I have not needed or even wanted to dual boot my Linux system in 5 years now, either for gaming or productive usages.

One additional feature I have not mentioned yet is for those that want to capture the VM for streaming/recording, etc. I have also written a native OBS plugin for Looking Glass that allows you to take the feed directly into OBS with no additional overheads. This then allows you to offload the video processing and encode to the host system (or another VM), even offloading to a GPU with hardware encode capability.
« Last Edit: August 06, 2022, 09:19:51 pm by gnif »
 
The following users thanked this post: Ed.Kloonk, JohanH, Bobson, evb149, RoGeorge, gnavigator1007, MK14, bd139, Nominal Animal, PKTKS, bill_c

Offline Halcyon

  • Global Moderator
  • *****
  • Posts: 5632
  • Country: au
Re: Windows VMs under Linux with Native Performance
« Reply #1 on: August 07, 2022, 06:13:08 am »
Brilliant work as always Geoffrey. I've plugged you on Linkedin.  :-+
 
The following users thanked this post: gnif

Offline PKTKS

  • Super Contributor
  • ***
  • Posts: 1766
  • Country: br
Re: Windows VMs under Linux with Native Performance
« Reply #2 on: August 07, 2022, 07:07:06 pm »
Passing here to drop the brilliant work mention as well...

But also to remind that  GPU pass and virtIO  VMs require a proper kernel compilation.
The options should allow the PCI and IOMMU specific options for that to happen.

Indeed an awesome step ahead  :-+
Paul
 
The following users thanked this post: gnif

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: Windows VMs under Linux with Native Performance
« Reply #3 on: August 07, 2022, 08:45:12 pm »
But also to remind that  GPU pass and virtIO  VMs require a proper kernel compilation.
The options should allow the PCI and IOMMU specific options for that to happen.

What do you mean? every major distro's stock kernel supports this and has done so for the past 5 years now.
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 23018
  • Country: gb
Re: Windows VMs under Linux with Native Performance
« Reply #4 on: August 07, 2022, 10:21:37 pm »
PKTKS is about 20 years behind the curve based on his other threads  :-DD

Interesting idea though. Will read into this tomorrow when I get some time. Either way more effort on this is appreciated so nice job  :-+
 
The following users thanked this post: gnif

Offline abquke

  • Regular Contributor
  • *
  • Posts: 128
  • Country: us
Re: Windows VMs under Linux with Native Performance
« Reply #5 on: August 07, 2022, 10:29:55 pm »
Can windows programs in VMs talk to hardware devices yet?

Software guys just dropped on me that they couldn't get Wine to use vendor software to talk to their USB widget... but we can't use windows because they couldn't access an I2C bus to control GPIO in windows...
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 23018
  • Country: gb
Re: Windows VMs under Linux with Native Performance
« Reply #6 on: August 07, 2022, 10:32:14 pm »
You can with virtualbox and the non free extensions from oracle. I was using it to do STM32 flash ages ago.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: Windows VMs under Linux with Native Performance
« Reply #7 on: August 07, 2022, 10:36:29 pm »
Can windows programs in VMs talk to hardware devices yet?

That's exactly what VFIO is.

Software guys just dropped on me that they couldn't get Wine to use vendor software to talk to their USB widget... but we can't use windows because they couldn't access an I2C bus to control GPIO in windows...

We commonly either pass-through a USB device to the VM (works mostly) or for a more reliable & complete solution you can pass through one of your USB controllers. If you don't have an extra controller you can pass through you can install a PCIe one and pass that into the VM. Controller pass-through gives 100% compatibility and native performance.

You can with virtualbox and the non free extensions from oracle. I was using it to do STM32 flash ages ago.

VirtualBox today is way behind the ball though, i'd suggest you use LibVirt which at it's core is QEMU/KVM and has far better support

As for STM32 flashing, you can just use the other FOSS project I started :)
https://sourceforge.net/p/stm32flash/wiki/Home/
It's packaged in most distros these days also.
« Last Edit: August 07, 2022, 10:41:46 pm by gnif »
 
The following users thanked this post: JohanH, MK14, bd139

Offline Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 4000
  • Country: au
  • Cat video aficionado
Re: Windows VMs under Linux with Native Performance
« Reply #8 on: August 07, 2022, 10:50:25 pm »
When I were a younger man, and virtualization was just a itch in Andy Groves' pants, I understood that virt would be good for sandboxing and security. I don't do gaming nor windows, but I don't understand how you could allow it access to the hardware and still trust it not betray your trust.
iratus parum formica
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: Windows VMs under Linux with Native Performance
« Reply #9 on: August 08, 2022, 03:04:05 am »
When I were a younger man, and virtualization was just a itch in Andy Groves' pants, I understood that virt would be good for sandboxing and security. I don't do gaming nor windows, but I don't understand how you could allow it access to the hardware and still trust it not betray your trust.

Read up on IOMMU, this isolates the hardware from your system preventing it from being able to compromise the system. Not only does it protect your system from malicious devices, it also can be used to protect your system from a malicious virtual machine that has access/use of the physical device. Under KVM it is a requirement to use IOMMU due to the obvious security concerns of running without it.
 
The following users thanked this post: Ed.Kloonk

Offline PKTKS

  • Super Contributor
  • ***
  • Posts: 1766
  • Country: br
Re: Windows VMs under Linux with Native Performance
« Reply #10 on: August 08, 2022, 08:48:34 am »
But also to remind that  GPU pass and virtIO  VMs require a proper kernel compilation.
The options should allow the PCI and IOMMU specific options for that to happen.

What do you mean? every major distro's stock kernel supports this and has done so for the past 5 years now.

I have stopped using so called stock kernels way back on mid 00s..

After a decade+ using these stock kernels and off the shelf distros.. it became an obvious problem that server setups have nothing in common with EDA and DAW workstations... and even less in common with gaming.. and absolute nothing with proxys..

So ... i made forks from LFS with my required specs..  things are several orders of magnitude SAFER  FAST and smaller in size..

So.   unless you plan to run a bloated messy crammed stuff..  you want a clean kernel..

My nephew game box runs stock stuff..

Not my servers or stations... decades for now
.

Can not stress how important it is
You can only realize that after having them on hand for own comparison

Paul
« Last Edit: August 08, 2022, 08:54:23 am by PKTKS »
 
The following users thanked this post: Bobson

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: Windows VMs under Linux with Native Performance
« Reply #11 on: August 08, 2022, 08:55:29 am »
After a decade+ using these stock kernels and off the shelf distros.. it became an obvious problem that server setups have nothing in common with EDA and DAW workstations... and even less in common with gaming..

So you have a low latency audio requirement, which is a realtime task, which requires a low latency kernel which impacts power usage and overall system throughput for most other tasks... got it.

So ... i made forks from LFS with my required specs..  things are several orders of magnitude SAFER  FAST and smaller in size..

Ok? so even if it's only one order of magnitude faster, that must mean a 3GHz system on your kernel would be equivalent to a 30GHz system? You can't make such broad claims... some aspect of your system for your use may be faster, but obviously not the entire system. As for size, so what? Size is only an issue if you're running on a MCU with limited memory.

So.   unless you plan to run a bloated messy crammed stuff..  you want a clean kernel..

Define clean? The kernel modules are only loaded as needed to support your hardware.

My nephew game box runs stock stuff..

No idea what this has to do with any of this.

Not my servers or stations... decades for now

You're preference, but not a requirement by any means. Again, also if you're running servers tuned for low latency/realtime type tasks, your servers overall throughput for all other tasks will be hampered.

Can not stress how important it is

Sorry, but I disagree to a large extent, please see/use Arch if you want to control every aspect of your system.

When it comes to running visualised environments for tasks as this thread is discussing your "faster", low latency kernels actually cause big issues due to the guest being starved of CPU time when it needs to service hardware interrupts.

Please be aware that I am actually an active kernel developer working on KVM and QEMU, along with the audio subsystems for Linux. With a stock kernel we can achieve latency numbers lower then windows running native on bare metal. At the end of the day it comes down to your use case and demands of your workload.
« Last Edit: August 08, 2022, 09:05:48 am by gnif »
 

Offline PKTKS

  • Super Contributor
  • ***
  • Posts: 1766
  • Country: br
Re: Windows VMs under Linux with Native Performance
« Reply #12 on: August 08, 2022, 09:00:05 am »
Exactly.

What the point of running qemu on a proxy?

Or having these options, IOMMU virtfs, in kernel running a dedicated workstation without qemu?

None.
It is useless and security hassle

Paul
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: Windows VMs under Linux with Native Performance
« Reply #13 on: August 08, 2022, 09:06:35 am »
Exactly.

What the point of running qemu on a proxy?

Or having these options, IOMMU virtfs, in kernel running a dedicated workstation without qemu?

None.
It is useless and security hassle

Paul

IOMMU protects your system from a faulty device that decides to use DMA to read/write RAM and crash your entire system. It applies to desktop and server environments and is not limited to virtualisation usage. Also it's a hardware feature of your CPU, not a kernel feature, you always have it, the kernel just needs to program the hardware so you get the free (no overheads, no latency increases) protection it offers.

What the point of running qemu on a proxy?

Huh? simple, don't install qemu  :palm:
« Last Edit: August 08, 2022, 09:11:05 am by gnif »
 
The following users thanked this post: bd139

Offline PKTKS

  • Super Contributor
  • ***
  • Posts: 1766
  • Country: br
Re: Windows VMs under Linux with Native Performance
« Reply #14 on: August 08, 2022, 09:11:37 am »
Actually it forces you to enable and compile a whole subset of options. .

Only usable while running a VM

All other  setups eg not VM, do not require this.. and require other fine tunes..

Probably one will only realize that by the moment you need one feature and not the others..

Paul
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: Windows VMs under Linux with Native Performance
« Reply #15 on: August 08, 2022, 09:16:33 am »
Actually it forces you to enable and compile a whole subset of options. .

Only usable while running a VM

Incorrect, the IOMMU is configured very early in the system boot by the kernel to provide device isolation and system protection, and it does not force you to use any VM options at all:

Code: [Select]
CONFIG_IOMMU_SUPPORT: IOMMU Hardware Support
General informations
The Linux kernel configuration item CONFIG_IOMMU_SUPPORT:

prompt: IOMMU Hardware Support
type: bool
depends on: CONFIG_MMU
defined in drivers/iommu/Kconfig
found in Linux kernels: 3.1–3.19, 4.0–4.20, 5.0–5.19, 5.19+HEAD

The only dependency is to have a MMU, which is a Linux kernel requirement.
IOMMU is not strictly a VM feature, but it can be leveraged in such configurations for added security.

Quote
In computing, an input–output memory management unit (IOMMU) is a memory management unit (MMU) that connects a direct-memory-access–capable (DMA-capable) I/O bus to the main memory. Like a traditional MMU, which translates CPU-visible virtual addresses to physical addresses, the IOMMU maps device-visible virtual addresses (also called device addresses or I/O addresses in this context) to physical addresses. Some units also provide memory protection from faulty or malicious devices.

An example IOMMU is the graphics address remapping table (GART) used by AGP and PCI Express graphics cards on Intel Architecture and AMD computers.

On the x86 architecture, prior to splitting the functionality of northbridge and southbridge between the CPU and Platform Controller Hub (PCH), I/O virtualization was not performed by the CPU but instead by the chipset.[1][2]
Source: https://en.wikipedia.org/wiki/Input%E2%80%93output_memory_management_unit

Disabling IOMMU on modern systems hurts performance AND system stabillity/security.

Edit: Don't confuse IO virtualization with Virtual Machines, they are very different things.

Edit2: Consider a device with a binary blob that needs loading to run it, lets say a Intel NIC, or a NVidia/AMD GPU which have closed firmware/drivers. IOMMU stops someone using the device as a mechanism to bypass system security either through intentional backdoors in the firmware/hardware, or bugs. There is no way in hell i'd ever consider putting a server online without IOMMU support enabled.
« Last Edit: August 08, 2022, 09:28:34 am by gnif »
 
The following users thanked this post: JohanH, bd139

Offline JohanH

  • Frequent Contributor
  • **
  • Posts: 620
  • Country: fi
Re: Windows VMs under Linux with Native Performance
« Reply #16 on: August 08, 2022, 09:42:35 am »
 :popcorn:
 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 2085
  • Country: sk
Re: Windows VMs under Linux with Native Performance
« Reply #17 on: August 08, 2022, 10:23:53 am »

I have stopped using so called stock kernels way back on mid 00s..


not using stock kernels coming with the distro made sense in the 90's but not in mid 00's. how do you handle security updates ?
i remember kernels with major number 0 , recompiling kernels was a pretty common routine back in the days but it simply makes no sense nowadays.
unless you develop an embedded system there is no reason to compile your kernel.. speed gain is next to none and security wise you put yourself into worse situation than using the stock kernel coming with the distro.

 
The following users thanked this post: gnif

Offline bd139

  • Super Contributor
  • ***
  • Posts: 23018
  • Country: gb
Re: Windows VMs under Linux with Native Performance
« Reply #18 on: August 08, 2022, 10:28:41 am »
PKTKS doesn't exist on the same planet as the rest of us.
 
The following users thanked this post: JPortici

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: Windows VMs under Linux with Native Performance
« Reply #19 on: August 08, 2022, 11:04:47 am »
Yeah, I had this also when I had an Athlon x64 and I wanted to experiment with the x86_64 support. I also had an EPOCH motherboard with the nForce2 chipset, which the kernel had no support for, but there were patches.

These days I only compile the kernel if I am doing development work on it, most of which is latency related testing or KVM debugging. Fixing AMDs bugs for them has also been a major cause of kernel re-compiles too, but to save time/effort I often resort to things like ftrace/kprobes for poking at PCI devices that are misbehaving.
 

Offline PKTKS

  • Super Contributor
  • ***
  • Posts: 1766
  • Country: br
Re: Windows VMs under Linux with Native Performance
« Reply #20 on: August 08, 2022, 01:07:35 pm »
There seems to be a misunderstanding ...

Combined with a necessary rant to insult and snob others (me included..)

I  *ALWAYS* configure the KERNEL  option IOMMU set as stated from above.
IT IS A NECESSARY COMPILE TIME OPTION SINCE version 2.x something.

However..
Please give me a break and just realize that

GPU passthrough technology (presenting PCI GPU to a virtual machine)  is also often known as IOMMU, although this is a bit of a misnomer, since the IOMMU is the hardware technology that provides this feature but also provides other features such as some DMA attacks  and etc_etall..


Sooooo   GPU passthrough  as intended (IOMMU ) and libvirt and friends are related to a QEMU CLIENT  machine  and will  require a proper HOST KERNEL ..  to enable  PCI passthrough IOMMU..
All options related to properly set the PCI passthrough  IOMMU

DON NOT MAKE MISTAKES ABOUT these 2 things... and start this sick insulting show...

Hope to shutup this nasty sick insults about things not related and degenerated...

I have reached such that age in life where insulting ... or having such freaking time with others just do not add anything to me...  I am a serious and very focused in intentions...

So spare me this nasty things... please.
Paul
« Last Edit: August 08, 2022, 01:12:26 pm by PKTKS »
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: Windows VMs under Linux with Native Performance
« Reply #21 on: August 08, 2022, 01:19:28 pm »
GPU passthrough technology (presenting PCI GPU to a virtual machine)  is also often known as IOMMU, although this is a bit of a misnomer, since the IOMMU is the hardware technology that provides this feature but also provides other features such as some DMA attacks  and etc_etall..


Sooooo   GPU passthrough  as intended (IOMMU ) and libvirt and friends are related to a QEMU CLIENT  machine  and will  require a proper HOST KERNEL ..  to enable  PCI passthrough IOMMU..
All options related to properly set the PCI passthrough  IOMMU

DON NOT MAKE MISTAKES ABOUT these 2 things... and start this sick insulting show...

Actually IOMMU is never called or confused with Passthrough, the term you're looking for is VFIO (Virtual Function IO) which makes use of IOMMU to map the device into a protective region to prevent a rogue VM from breaking/crashing or exploiting the host (VM escape). I mean, the kernel module use for this is literally called `vfio_pci`. There is a reddit community for VFIO of over 33k users (https://www.reddit.com/r/VFIO/), and a Discord community called VFIO of over 4.6k users.  Futher to this, VFIO is not limited to use in Virtual Machines, it has user-space access applications bypassing the kernel in special circumstances with custom devices.

You can actually perform VFIO pass-through without IOMMU support at all, but it is dangerous to do so and as such by default the KVM kernel module will refuse to do so unless patched. For example, the ACS override patch fools the kernel into believing that the PCIe devices are all in isolated IOMMU groups when they may not be.

As for your argument of disabling these features to make a kernel faster or more secure, this is just simply not true. Kernel modules are only loaded as they are needed either by some feature you want (like KVM) or a device that requires the module to operate, otherwise they sit idle on disk. The VM features exist in the `kvm` kernel module and all major distros compile these as modules, they are not inlined into the kernel. Unless you have actually modprobed these kernel modules then you have a clean kernel that is not full of bloat you don't need.

From a security point of view, If an attacker on your system has the permissions needed to insmod/modprobe a kernel module, this means they have `root` access and it doesn't matter what kernel you run, or how you built it, your security has already been defeated.
« Last Edit: August 08, 2022, 01:22:32 pm by gnif »
 
The following users thanked this post: macboy

Offline PKTKS

  • Super Contributor
  • ***
  • Posts: 1766
  • Country: br
Re: Windows VMs under Linux with Native Performance
« Reply #22 on: August 08, 2022, 01:22:46 pm »
Actually IOMMU is never called or confused with Passthrough, the term you're looking for is VFIO (Virtual Function IO) which makes use of IOMMU to map the device into a protective region to prevent a rogue VM from breaking/crashing or exploiting the host (VM escape). I mean, the kernel module use for this is literally called `vfio_pci`.
(..)

As you wish my dear...  by now  you just got the misunderstanding ..

I have no intention to lecture insult or or go further..

Pretty sure you got the point between the 2 uses of IOMMU...

Paul
 

Offline krokodyl1220

  • Contributor
  • Posts: 17
  • Country: pl
Re: Windows VMs under Linux with Native Performance
« Reply #23 on: August 10, 2022, 03:23:56 pm »
Ranting and discussion aside, I am another one of happy people using LookingGlass. In my case it is to run ECAD and MCAD and it works flawlessly. Thank you!

Recently I have created a simple script to help when using VMs in user session mode (I dislike running VMs as root on my workstation) to isolate the CPUs with cgroups. If you want to check it out see: https://gitlab.com/krokodyl1220/vfio-vm-tools

Unfortunately for PKTKS it uses systemd :D
 
The following users thanked this post: gnif, bd139

Offline Marco

  • Super Contributor
  • ***
  • Posts: 6694
  • Country: nl
Re: Windows VMs under Linux with Native Performance
« Reply #24 on: August 10, 2022, 05:11:11 pm »
How much memory do you use in the laptop to get decent performance?
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf