The following is a small collection of tweaks for both host and guest systems collected over time.

The host machine will be the machine that hosts the virtual machines.

If the host is a Linux system, the following changes may improve performance.

• Use the more elaborate block MQ for all SCSI devices. This can be achieved by appending scsi_mod.use_blk_mq=y dm_mod.use_blk_mq=y to /etc/default/grub as kernel parameter.
• In case the system allows for NUMA balancing, ie:

dmesg | grep NUMA

shows any available nodes, then consider appending numa_balancing=enable to the kernel boot parameters. NUMA balancing will allow libvirt to balance the virtual machines across processors for better performance and, hopefully, for a more uniform load spread.

The rule is to always prefer virtio drivers over emulated drivers since virtio drivers are aware of the virtualised environment and will invariably lead to better performance. You can browse the libvirt FUSS page for various instances where virtio drivers can be used (networking, storage, etc…).

The guest machine will be a virtual machine running on top of the host operating system. General guidelines can be mentioned here:

• Avoid using QCOW/2 or file-based virtual machines (even RAW) if the guest is meant to be always on. Although the QCOW image format can offer some nice features, an LVM backed storage medium on a physical disk will always outperform (considering both speed and convenience).

If the guest is a Linux system, the following changes may improve performance inside virtual machine.

• zswap is a very cool technology that allows transparently compressing pages in RAM and using them instead of disk-based swap. For instance, the following kernel parameters added to /etc/default/grub:

zswap.compressor=lz4 zswap.max_pool_percent=50

will use the very fast LZ4 compression algorithm and split off half the RAM to be used as compressed swapspace. One of the side-benefits is cheating in leased virtual machine containers such as the ones provided by DigitalOcean.

• The default setting for hugepages in recent kernels is to always attempt to use hugepages. Unfortunately, depending on your workload, hugepages may consume way more than was intended given programs that do not need to allocate large chunks of memory. A consequence thereof is that the guest may end up eating much more RAM than necessary. To set hugepages to only be used when requested, set the following kernel parameter transparent_hugepage=madvise.

• Use the noop elevator inside guest machines: there is no need for complicated schedulers; if any at all since libvirt will be taking care of all requests on the host side. Append elevator=noop to the kernel boot parameters to disable CFQ. Contrary to the host system, using the new block MQ is not necessary and a very simple, no-operation scheduler such as noop is perfectly suited. When using the noop elevator, make sure to top off with disabling the guest disk cache in the libvirt configuration.

• Disable the USB stack by appending usbcore.nousb to the kernel parameters. If you are virtualizing servers, then there is really no need to have a loaded USB stack, especially since USB hardware is known to generate large number of interrupts that are really not welcome in a virtualization scenario.

The last known to (reasonably TM) work Windows release as a libvirt guest is Windows 7 and any followup operating systems (currently up and including Windows 10) have severe issues with libvirt. Mostly, various sources claim that the issue lies with huge amounts of context switches produced by high-resolution timers embedded in the Windows operating system or other server software such as Microsoft SQL Server (MSSQL). Otherwise, Windows 10 (and by that, we mean, the monstrosity that comprises the kernel and the GUI) is full of bloatware, ie: Office 365 spam popup offers that, albeit benign, still produce background events that the CPU will just have to handle.

In particular, MSSQL is known to hold an internal timer that fires every in order to provide "high performance timing". Sadly, the consequence is that the CPU spools up to excess under libvirt. The MSSQL timer can be disabled using a trace flag.

Memory ballooning has been known to impact performance when used in conjunction with Windows guests. Simply avoid installing the memory balooning driver on Windows and ignore the feature until a solution has been found.

Finally, if Windows 10 is a must, then seriously consider replacing the "Metro" interface along with the entire shell with some lightweight variant - for instance, LiteStep and other alternatives. It is scary how much the shell in Windows 10 impacts performance - most likely, since it seems tied to a lot of tertiary features that just generate background events meant for attended desktop machines (ie: hipster laptop machines where you really want to find out the very latest Tweet and not a Windows machine that you need to run (mostly) unattended).