User avatar
gmol1
Posts: 62
Joined: Fri Dec 04, 2015 12:33 am

[Guide] VGA Passthrough (OUTDATED!)

Sat Jun 04, 2016 9:58 am

Hey there guys, this guide is outdated, i'll try to update it in the next couple of weeks but I cant promise anything since I don't own a desktop anymore.


Now that the VFIO-PCI module is available we can easily pass our PCI devices to QEMU/KVM so I decided to make a guide, however before getting into the details there are a couple of important things to keep in mind:

What we will do is to create a dracut module in order to bind the vfio-pci driver to our GPU, however, apparently there's a problem with this and that's that the module should be wiped in the next dracut update or at least that's what Ikey (Solus developer) has told me.

Now, there should be several ways to achieve the same result but there are a couple of reasons of why I decided to take this path. The main reason is that my GPUs are identical so that means they have the same PCI vendor and device ID making it impossible for me to make vfio-pci or pci-stub claim one of the cards without claiming both and another reason is that I can't get modprobe to execute scripts for some reason... However i'll try to get another GPU next week so I can expand the options.

Another thing to keep in mind is that I haven't tried PCI passthrough with Intel CPUs, so everything in this guide related to Intel is just theorical until someone tries it.

If anyone is willing to spend time with me to test other methods, send me a pm.

With things clear, lets begin:

The first thing you have to do is to check if your CPU supports virtualization and IOMMU, if it does, enable both options in the BIOS. Once enabled you can set up things either automatically or manually, both ways are described below, choose the one that suits you best:

Automatically
Download the script that will do the dirty work:

Code: Select all

$ git clone https://github.com/gmol1/vfio-bind.git
or 
$ wget https://github.com/gmol1/vfio-bind/archive/master.zip
Next give the script execution privileges and run it (it will check if IOMMU is enabled and working correctly):

Code: Select all

$ chmod +x vfio-bind.sh
$ sudo ./vfio-bind.sh
Select the device that you want to pass, wait for the script to finish and reboot.

Now run "lspci -nnk" and you should see that the driver in use for the device is vfio-pci:

Code: Select all

$ lspci -nnk
...
04:00.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI Cypress [Radeon HD 5800 Series] [1002:6899]
	Subsystem: VISIONTEK Device [1545:1002]
	Kernel driver in use: vfio-pci
04:00.1 Audio device [0403]: Advanced Micro Devices [AMD] nee ATI Cypress HDMI Audio [Radeon HD 5800 Series] [1002:aa50]
	Subsystem: VISIONTEK Device [1545:aa50]
	Kernel driver in use: vfio-pci
...
You can now move to the QEMU Setup section.

Manually
Check if IOMMU is enabled:

In the case of AMD, IOMMU is enabled by default but in order to prevent possible i/o page faults, open the file "/etc/default/grub", find the line "GRUB_CMDLINE_LINUX_DEFAULT", add "iommu=pt", save, update grub by running "sudo update-grub" and reboot. Then run "dmesg | grep -e AMD-Vi" to check if everything is working and you should get something similar to this:

Code: Select all

$ dmesg | grep -e AMD-Vi
[    1.357896] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.357897] AMD-Vi: Interrupt remapping enabled
[    1.358007] AMD-Vi: Lazy IO/TLB flushing enabled
In the case of Intel you will have to enable IOMMU in the kernel. Open the file "/etc/default/grub", find the line "GRUB_CMDLINE_LINUX_DEFAULT", add "intel_iommu=on", save, update grub by running "sudo update-grub" and reboot.

Now run "dmesg | grep -e DMAR -e IOMMU" and if IOMMU is enabled you should get something like this:

Code: Select all

$ dmesg | grep -e DMAR -e IOMMU
...
[    0.000000] Intel-IOMMU: enabled
...

or

$ dmesg | grep -e DMAR -e IOMMU
...
[    0.000000] DMAR: IOMMU enabled
...
Now we have to check that IOMMU actually works, we do this by checking the IOMMU groups, run:

Code: Select all

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done
In my case I have two identical cards, so I get this:

Code: Select all

...
IOMMU group 16
	01:00.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI Cypress [Radeon HD 5800 Series] [1002:6899]
	01:00.1 Audio device [0403]: Advanced Micro Devices [AMD] nee ATI Cypress HDMI Audio [Radeon HD 5800 Series] [1002:aa50]
...
IOMMU group 19
	04:00.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI Cypress [Radeon HD 5800 Series] [1002:6899]
	04:00.1 Audio device [0403]: Advanced Micro Devices [AMD] nee ATI Cypress HDMI Audio [Radeon HD 5800 Series] [1002:aa50]
As you can see I have only 2 devices in each group and each device corresponds to the GPU and the Audio device of the graphics card and this is how you should have it too. If you have other devices grouped with your graphics card then its possible that the PCIe slot doesn't support isolation properly, so try switching the graphics card from PCIe slot, if the issue persists you will have to pass every device in the group to the VM if you're fine with that.

So, now that we have IOMMU working, we have to create a Dracut module. What for? Well, we need to "hijack" the GPU and bind the vfio-pci driver before the Radeon/Nouveau driver is assigned to it, otherwise if you try to unbind the driver from the GPU things will go bad, very bad (at least with Radeon). So, create the directory "40vfio-bind" in "/usr/lib/dracut/modules.d/":

Code: Select all

$ sudo mkdir /usr/lib/dracut/modules.d/40vfio-bind
Next, populate the directory:

Code: Select all

sudo touch /usr/lib/dracut/modules.d/40vfio-bind/module-setup.sh
sudo touch /usr/lib/dracut/modules.d/40vfio-bind/vfio-bind.sh
Edit the "module-setup.sh" file with your favorite text editor and add:

Code: Select all

#!/bin/bash
# -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
# ex: ts=8 sw=4 sts=4 et filetype=sh
check() {
return 0
}

depends() {
return 0
}

install() {
inst_hook pre-trigger 91 "$moddir/vfio-bind.sh"
}

installkernel() {
instmods vfio vfio_iommu_type1 vfio_pci vfio_virqfd
}
Now you must identify the devices that you want to bind, to do so run "lspci -D":

Code: Select all

$ lspci -D
...
0000:01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cypress [Radeon HD 5800 Series]
0000:01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cypress HDMI Audio [Radeon HD 5800 Series]
0000:02:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
0000:03:00.0 IDE interface: Marvell Technology Group Ltd. 88SE9172 SATA III 6Gb/s RAID Controller (rev 11)
0000:04:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cypress [Radeon HD 5800 Series]
0000:04:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cypress HDMI Audio [Radeon HD 5800 Series]
...
In my case I want to bind my secondary graphics card so I'll take "0000:04:00.0" and "0000:04:00.1" and add the following to "vfio-bind.sh":

Code: Select all

#!/bin/sh

DEVS="0000:04:00.0 0000:04:00.1"

for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci
We're almost done, now we have to regenerate initramfs by running "dracut -f --kver `uname -r`" and reboot.

If everything went right you should now see that the driver that the device is using is vfio-pci:

Code: Select all

$ lspci -nnk
...
04:00.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI Cypress [Radeon HD 5800 Series] [1002:6899]
	Subsystem: VISIONTEK Device [1545:1002]
	Kernel driver in use: vfio-pci
04:00.1 Audio device [0403]: Advanced Micro Devices [AMD] nee ATI Cypress HDMI Audio [Radeon HD 5800 Series] [1002:aa50]
	Subsystem: VISIONTEK Device [1545:aa50]
	Kernel driver in use: vfio-pci
...
Now we just have to setup QEMU.

QEMU Setup
We're almost there, install "qemu" and "virt-manager":

Code: Select all

$ sudo eopkg it qemu virt-manager -y
Enable "libvirtd":

Code: Select all

$ systemctl enable libvirtd
And reboot. Now its time to create our Virtual Machine, open virt manager by running "virt-manager" and go to:

Code: Select all

"File" -> "New Virtual Machine" -> "Local install media (ISO image or CDROM)"
Select your media, press forward and choose how much ram do you want, press forward and pay attention, this part is important:

Code: Select all

-> "Select or create custom storage" -> "Manage..." 
-> Add a new volume by pressing the green "+" button to the right side of the "Volumes" label 
-> Select "raw" as format, change the allocation to 0 -> "Finish"
-> "Choose Volume" or double click the volume
-> "Forward", check "Customize configuration before install" -> "Finish"
Now download the VirtIO drivers and:

Code: Select all

-> Select "IDE Disk 1" -> Press "Advanced options" -> Change "Disk bus" to "SCSI" -> "Apply"
-> Select "Controller SCSI" -> Change "Model" to "VirtIO SCSI" -> "Apply"
-> "Add Hardware"-> Storage -> Change "Device type" to "CDROM device" -> "Manage..." -> Locate the VirtIO ISO -> "Finish"
-> "Add Hardware" -> "PCI Host Device" -> Select your device (x2)
Press "Begin Installation", load the drivers when asked and install.

If everything went as planned you should now see the GPU listed in the "Device Manager", proceed to install the drivers and enjoy.

Optimizations (Soon™)
Last edited by gmol1 on Wed May 31, 2017 9:51 pm, edited 3 times in total.

User avatar
Justin
Development Team
Posts: 3613
Joined: Mon Jul 28, 2014 10:42 am
Location: Adelaide, Australia
Contact: Website Google+ Twitter

Re: [Guide] VGA Passthrough

Sat Jun 04, 2016 1:29 pm

Wow that is one insane guide. I wish my CPU had IOMMU. Upon my next build I will be definitely trying this. If nobody has helped with the 2 different card guide by then I will certainly try with your assistance!

User avatar
gmol1
Posts: 62
Joined: Fri Dec 04, 2015 12:33 am

Re: [Guide] VGA Passthrough

Sun Jun 05, 2016 1:46 am

Thanks :) hopefully more people will get involved so we can improve it.

User avatar
linuxhelmet
Posts: 31
Joined: Wed Jun 22, 2016 2:59 pm

Re: [Guide] VGA Passthrough

Wed Jun 22, 2016 5:24 pm

Awesome guide! If it wasn't for the smaller package base for Solus I would have tried this on mine already! Currently have Deepin as the host (Debian SID) and have run pretty much everything Debian-based (Ubuntu spins) but maybe I'll give this a shot in the near future :D

User avatar
Justin
Development Team
Posts: 3613
Joined: Mon Jul 28, 2014 10:42 am
Location: Adelaide, Australia
Contact: Website Google+ Twitter

Re: [Guide] VGA Passthrough

Thu Jun 23, 2016 9:01 am

What is missing from Solus that you use on Debian?

User avatar
gmol1
Posts: 62
Joined: Fri Dec 04, 2015 12:33 am

Re: [Guide] VGA Passthrough

Thu Jun 23, 2016 8:04 pm

linuxhelmet wrote:Awesome guide! If it wasn't for the smaller package base for Solus I would have tried this on mine already! Currently have Deepin as the host (Debian SID) and have run pretty much everything Debian-based (Ubuntu spins) but maybe I'll give this a shot in the near future :D
Thanks and I agree the package availability is a big problem, once I get some free time i'll update this guide and start working on a package manager which will kind of follow the concept of Homebrew so we can have way more packages available.

User avatar
linuxhelmet
Posts: 31
Joined: Wed Jun 22, 2016 2:59 pm

Re: [Guide] VGA Passthrough

Thu Jun 23, 2016 8:59 pm

Justin there's a ton of free games available on Debian that can be installed outside of Steam. Also Debian SID rolls. So kernel wise it's always current and I'm always getting the bleeding edge packages for my KVM VGA passthrough setup.

Example is the newest version of virt-manager allows you to use the passthrough feature by clicking a checkbox rather than editing the virsh file. This way all my stuff is current and I don't have to completely install new versions of a distro or worry about "upgrading" versions (Ex: 15.04 to 15.10) without something breaking. This is why I left Ubuntu and forks.

As for Solus I'm considering it. From what I hear though it's not fully rolling but rather a semi-rolling setup. The packages roll but the kernel doesn't. The problem with this is that the passthrough setup relies on some updates from the kernel for it to work and gains improvements with each release.

Not trying to dismay anyone from trying this btw! The speed of Solus should greatly improve boot times for passthrough as well as decrease any lag. (mine takes over a minute to boot with SSD!) The only reason I'm even debating switching is setting up passthrough setups is a pain. So if Solus goes bonkers or if an upgrade to a newer build fails I have to redo all that work. Speed (Solus is faster) vs Rebuild time (almost nonexistent on Debian)

/rant.

Also Homebrew? Kinda like the homebrew channel? O.o

User avatar
gmol1
Posts: 62
Joined: Fri Dec 04, 2015 12:33 am

Re: [Guide] VGA Passthrough

Fri Jun 24, 2016 4:25 am

The virt-manager version available in the repo is 1.3.2 (which was the latest until 5 days ago when 1.4.0 got released), so you can assign PCI devices from it. Also I don't know what you mean with this:
As for Solus I'm considering it. From what I hear though it's not fully rolling but rather a semi-rolling setup. The packages roll but the kernel doesn't. The problem with this is that the passthrough setup relies on some updates from the kernel for it to work and gains improvements with each release.
The Solus kernel already has all the needed patches for passthrough but in any case you needed a particular module or patch applied to it, you can ask for and the they will probably enable/patch it.
Also Homebrew? Kinda like the homebrew channel? O.o

Like Homebrew the package manager, there's also Linuxbrew which is a Homebrew fork but many precompiled binaries won't work in Solus + they won't use the optimized Solus libraries and most of the time you won't be able to compile from source due to differences between the tools.

User avatar
Justin
Development Team
Posts: 3613
Joined: Mon Jul 28, 2014 10:42 am
Location: Adelaide, Australia
Contact: Website Google+ Twitter

Re: [Guide] VGA Passthrough

Sat Jun 25, 2016 12:18 am

We try and keep the latest applications so you should never be behind in those. As for the kernel stuff, we follow the LTS branch currently however there has been discussion of the ability to have multiple kernels in the repository, ie an LTS + a current stable for instance. I believe this relies a lot on some tweaking to be done to our boot manager to allow it though, but keep your eyes peeled :)

As gmol1 said, if there's a commit in a newer kernel that we don't have we may be able to backport it, should it be beneficial to users.

Zackptg5
Posts: 6
Joined: Tue Feb 14, 2017 4:25 pm

Re: [Guide] VGA Passthrough

Fri Feb 17, 2017 11:08 pm

So is this not possible with a uefi setup since it doesn't use grub? I have an intel cpu and so I would need to enable IOMMU in the kernel

Return to “Tutorials”