User avatar
Sori
Posts: 3
Joined: Mon Aug 06, 2018 11:47 pm

[Guide] Basic VFIO Passthrough on Solus

Mon Aug 06, 2018 11:58 pm

This is intended to help anyone needing to setup VFIO passthrough. This guide assumes you know what VFIO passthrough is and the hardware requirements for it, have two graphics cards, and know how to use virt-manager.

Make sure IOMMU is enabled (IOMMU is enabled by default):

Code: Select all

dmesg | grep IOMMU
Intel output flat out says IOMMU is enabled, AMD says "IOMMU performance counters supported"

If IOMMU isn't enabled, you can set it in your BIOS and then use a kernel parameter.
Intel:

Code: Select all

echo "intel_iommu=on" >> /etc/kernel/cmdline.d/20_vfio.conf
AMD:

Code: Select all

echo "amd_iommu=on" >> /etc/kernel/cmdline.d/20_vfio.conf
Use this script to make sure what you want to passthrough isn't in the same IOMMU group as something else you want to passthrough:

Code: Select all

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
   n=${d#*/iommu_groups/*}; n=${n%%/*}
   printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;
If you have things you don't want to pass through in the same IOMMU group, run this:

Code: Select all

echo "pcie_acs_override=downstream" >> /etc/kernel/cmdline.d/20_vfio.conf
Run these commands to add the vfio configuration to boot:

Code: Select all

echo 'force_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"' >> /etc/dracut.conf.d/vfio.conf    
Run this to find your device IDs:

Code: Select all

lspci -nnk
Find the device you want to pass through, and look for the IDs near the end. This is the device I'm passing through:

Code: Select all

27:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X] [1002:67df] (rev c7)
   Kernel driver in use: amdgpu
27:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
   Kernel driver in use: snd_hda_intel
In my case, the device IDs are 1002:67df and1002:aaf0. Find yours, and replace my IDs with yours. Also make note of the drivers in use as you'll need those later. Also make note of the PCI IDs as you might need them. Mine are 27:00.0 and 27:00.1

Run these, replacing my IDs with yours to add the kernel parameters:

Code: Select all

echo "rd.driver.pre=vfio-pc" >> /etc/kernel/cmdline.d/20_vfio.conf   
echo "vfio-pci.ids=1002:67df,1002:aaf0" >> /etc/kernel/cmdline.d/20_vfio.conf  
Run this to add the IDs via modprobe:

Code: Select all

echo "options vfio-pci ids=1002:67df,1002:aaf0" >> /etc/modprobe.d/vfio.conf   
And then this to make sure the VFIO drivers load before the standard ones (replace my drivers with the ones you need):

Code: Select all

echo "softdep amdgpu pre: vfio vfio_pci" >> /etc/modprobe.d/vfio.conf   
echo "softdep snd_hda_intel pre: vfio vfio_pci" >> /etc/modprobe.d/vfio.conf
Run these to refresh the boot manager and kernel image:

Code: Select all

dracut -f --kver 'uname -r'
clr-boot-manager update
Then reboot.

You can test to make sure the driver is bound by running this command and making sure the kernel driver says vfio:

Code: Select all

lspci -nnk
If it doesn't you can create module to override the driver as well:

Code: Select all

echo 'add_dracutmodules+="vfio-bind"' >> /etc/dracut.conf.d/vfio.conf  
Create this file: /usr/lib/dracut/modules.d/40vfio-bind/module-setup.sh

Code: Select all

#!/bin/bash
# -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
# ex: ts=8 sw=4 sts=4 et filetype=sh
check() {
return 0
}

depends() {
return 0
}

install() {
inst_hook pre-trigger 91 "$moddir/vfio-bind.sh"
}

installkernel() {
instmods vfio vfio_iommu_type1 vfio_pci vfio_virqfd
}
Create this file - replacing the "DEVS" with the PCI IDs listed previously in lspci: /usr/lib/dracut/modules.d/40vfio-bind/vfio-bind.sh

Code: Select all

#!/bin/sh
DEVS="0000:27:00.0 0000:27:00.1"

for DEV in $DEVS; do
     echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci
Do this and then reboot:

Code: Select all

dracut -f --kver 'uname -r'
clr-boot-manager update 
It should be bound now. Check again with lspci -nnk

Install virt-manager - set up your VM and configure it to pass through the graphics card and anything else you want to pass through. Don't boot up the VM.

Edit the resulting xml file under /etc/libvirt/qemu/ and we're going to make some changes for performance.

I have a Ryzen 5 1600, so a 6 core CPU and my CPU pinning is set to that. Adjust yours for your CPU:

Delete the vcpu placement , and the cpu fields and replace with this:

Code: Select all

<vcpu placement='static'>6</vcpu>
<cputune>
   <vcpupin vcpu='0' cpuset='1'/>
   <vcpupin vcpu='1' cpuset='5'/>
   <vcpupin vcpu='2' cpuset='2'/>
   <vcpupin vcpu='3' cpuset='6'/>
   <vcpupin vcpu='4' cpuset='3'/>
   <vcpupin vcpu='5' cpuset='7'/>
  <emulatorpin cpuset='0,4'/>
 </cputune>
<cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='3' threads='2'/>
</cpu>
If you want to pass a disk through, this is what it looks like. Edit as needed:

Code: Select all

<disk type='block' device='disk'>
    <driver name='qemu' type='raw'/>
    <source dev='/dev/sda'/>
    <target dev='vdb' bus='sata'/>
   <address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
Run this to pass the changes to virt-manager:

Code: Select all

virsh define <XML FILE>
Go ahead and boot and install Windows per normal. Mine has been exceptionally stable and I can't tell the difference between using the VM and using a bare metal windows install.

Check out the follow up post to make your life easier: viewtopic.php?f=11&t=12597

Return to “Tutorials”