Table of Contents

About

These notes describe a minimal setup where the GPU card of a host system is passed onto a virtualized guest with the purpose of providing acceleration for the hardware transcoding of video files to be used with PVRs such as Plex Media Server or Jellyfin.

Hardware Overview

One of the interesting applications for GPU passthrough can be found with environments where the host machine is used solely for virtualization but still benefits from some hardware that would not end up being use at all.

Our current setup is a minimal BeeLink T4 server, with an Intel HD Graphics 500 GPU that is still useful enough to perform hardware transcoding via QuickSync for a virtualized machine that runs the Plex Media Server.

In this particular case, the setup will involve removing the Intel HD Graphics 500 GPU from the host operating system and passing it through to the guest operating system.

Hardware Requirements

The following CPU extensions are needed and should be enabled inside the BIOS of the host system:

Host Setup

Setting up the host system involves enabling IOMMU, using VFIO such that the GPU can be replaced by a different driver and disabling VGA. The following are the parameters that should be added to the kernel command line: 

kvm.ignore_msrs=1 intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 vfio_pci.disable_vga=1 vfio_pci.disable_idle_d3=1 vfio-pci.ids=8086:5a85

where:

In order to determine the VFIO PCI ids, use lspci to determine the ids of the GPU that should be passed to the guest: 

lspci -nn

In this case, the matching GPU is an Intel HD Graphics 500: 

00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 500 [8086:5a85] (rev 0b)

where 8086:5a85 corresponds to the id passed to the vfio-pci.ids= kernel parameter.

Next, the VFIO modules should be loaded early on boot, preferrably even before any GPU modules are loaded and in order to do that the following modules are added to initramfs by editing /etc/initramfs-tools/modules and appending the following lines: 

vfio_pci
vfio
vfio_iommu_type1

followed by running the commands: 

update-initramfs -u
update-grub

for the changes to be committed.

After a reboot, virsh should be used to locate the PCI device to be passed to the guest by issuing the command: 

virsh nodedev-list --cap pci

that will list a number of PCI addresses such as: 

pci_0000_00_00_0
pci_0000_00_02_0
pci_0000_00_03_0
pci_0000_00_07_0
pci_0000_00_10_0
pci_0000_00_10_1
pci_0000_00_14_0
pci_0000_00_14_1
pci_0000_00_14_2
pci_0000_00_14_3
pci_0000_00_19_0
pci_0000_00_1a_0
pci_0000_00_1a_1
...

Each of these addresses can then be further queried, for example, to query the second address on the list pci_0000_00_02_0, issue the command: 

virsh nodedev-dumpxml pci_0000_00_02_0

and it should result in some output such as the following: 

<device>
  <name>pci_0000_00_02_0</name>
  <path>/sys/devices/pci0000:00/0000:00:02.0</path>
  <parent>computer</parent>
  <driver>
    <name>vfio-pci</name>
  </driver>
  <capability type='pci'>
    <class>0x030000</class>
    <domain>0</domain>
    <bus>0</bus>
    <slot>2</slot>
    <function>0</function>
    <product id='0x5a85'>HD Graphics 500</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <iommuGroup number='0'>
      <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </iommuGroup>
    <pci-express/>
  </capability>
</device>

Now it can be seen that the Intel HD Graphics GPU has been found and that its address description is: 

<address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

which is what is needed in order to change the domain file such that libvirt picks up the GPU.

All that remains to be done for the host is to edit the domain description for the virtual machine and pass the GPU through. Here is the snippet that gets add to the devices section of the domain XML file relevant to the GPU passthrough: 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>

where:

Guest Setup

If everything went well, the GPU should be observed as being recognized by the Linux kernel and that can be checked using dmesg and with lspci in order to confirm that the GPU has been attached.

When the GPU is passed through from the host to the guest, the guest perceives the GPU as a new device that will need drivers in order to be useful. That being said, the GPU drivers have to be installed on the guest as well as the host. For this tutorial, the Intel HD Graphics drivers are to be found on Debian as part of the firmware-misc-nonfree package: 

aptitude install firmware-misc-nonfree

Some additional utilities can be installed such as Intel GPU tool to monitor the utilization of the GPU, for instance by installing the intel-gpu-tools package on debian: 

aptitude install intel-gpu-tools

This will give access to the intel_gpu_top command that will display the current GPU utilization and the process that is currently accessing the GPU. For instance, using Plex Media Server and playing a movie whilst forcing Plex to transcode the videos, the following utilization is displayed by intel_gpu_top:

Plex Media Server Setup

Plex Media Server has to be setup, at least for testing, to disable "direct play", where the server would send the file directly to the browser, and to use hardware transcoding instead. This can be accomplished by accessing the Plex settings and:

playing with the hardware transcoding might achieve various effects where the passed through GPU might only be used partially, even only for audio remixing, or fully by transcoding the entire stream.