The following command creates a new Virtual Machine with the name "live", 1024MB of RAM, boots from the live.iso in the current path, opens up an available VNC port listening on 192.168.1.1 and will use the pre-allocated image file /var/lib/libvirt/images/live.img as a harddrive.

To pre-allocate the image file run:

truncate --size=8192M /var/lib/libvirt/images/guest.img

which will create an image file at /var/lib/libvirt/images/guest.img of 8GB size.

Next, to start the install, run:

virt-install --name live \
    --network bridge:br0 --ram 1024 \
    --boot cdrom --cdrom live.iso \
    --graphics vnc,listen=192.168.1.1,port=5901 \
    --disk /var/lib/libvirt/images/live.img

destroy live #halts the machine forcibly
undefine live #removes live from the list of machines managed by virsh

First attach to the domain and edit it, by issuing:

virsh -c qemu:///system
list --all
edit domain # edit the domain name, obtained from the previous command

and then insert the following snippet:

    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/path/to/file.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>

where /path/to/file.iso is the path to the ISO file to attach to the domain.

virtio is a thin-layer networking and disk driver that will skip the overhead of a full emulation thereby making the guest faster. In order to enable it, edit a domain and change the disk section such that they use the virtio driver:

    <disk type='file' device='disk'>
...
      <target dev='hda' bus='virtio'/>
...
    </disk>

and be sure to remove any address tag because virsh will recreate it.

Also change the network driver to virtio:

    <interface type='bridge'>
...
      <model type='virtio'/>
      <driver>
        <host gso='off' tso4='off' tso6='off' ecn='off' ufo='off'/>
        <guest tso4='off' tso6='off' ecn='off'/>
      </driver>
    </interface>

The driver block ensures that various offload parameters are turned off for the host and guest which is deemed to be good practice for virtio interfaces on a bridge. Additionally, the bridge can be configured to turn off most of the offloading:

ethtool -K br0 tso off sg off ufo off gro off gso off

where:

• br0 is the bridge interface to which the virtio interfaces are added.

Multi-queue virtio-net provides an approach to networking that scales up well as the number of virtual CPUs increases by allowing packets to be transmitted and received in parallel.

To configure multi-queue virtio-net, edit the guest domain configuration and add:

<driver name='vhost' queues='N'/>

to the network interface where N is the number of queues (from 1 to 8).

For instance, the following configuration:

    <interface type='bridge'>
      <mac address='5b:48:e3:4a:5b:32'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
    </interface>

will switch to the vhost driver and use 2 queues.

Native asynchronous IO gives better performance for virtual images, it can be enabled by editing the domain and adding the attribute io='native' to the disk tag:

<driver name='qemu' type='raw' io='native'/>

Add the cache='none' attribute to the driver tag of the domain:

<driver name='qemu' type='raw' io='native' cache='none'/>

Execute:

virsh capabilties

in order to list all the available CPU features. The CPU type with all the features are listed under the <cpu> tag and can be copied directly to a domain configuration. In order to do that, first connect to the system using virsh:

virsh -c qemu:///system

and then edit the domain that the features should be added to:

edit guest1

and copy the desired features, including the tags (<cpu>…</cpu>) from the first step into the configuration and save the file.

If you wish to expose the entire set of features, you can use the passthrough parameter passed to cpu by editing the guest configuration:

<cpu mode='host-passthrough'>
</cpu>

Sometimes the default settings do not select the correct topology. This can be observed by running:

ps aux | grep kvm

and reading the following line:

-smp 2,sockets=1,cores=1,threads=1

Comparing to the correct topology, which can be obtained by running:

virsh capabilities | grep topology

which is:

<topology sockets='1' cores='2' threads='1'/>

we can observe that the number of cores is detected wrongly.

Thus, to enable the correct topology, we take the output of the previous command and place it in the cpu subsection of the domain xml:

  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>

and then shutdown and restart the guest.

vnet devices created by virsh perform better with a txqueuelen set to a higher txqueuelen. In order to test, the txqueuelen can be set manually using:

ifconfig vnet0 txqueuelen 2500

To make the changes persistent, we add a file called 60-vnet.rules to /etc/udev/rules.d containing the line:

KERNEL=="vnet[0-9]*", RUN+="/sbin/ip link set %k txqueuelen 2500"

You may find an error message indicating that the current clock-source is unstable. If you issue:

dmesg | grep TSC

you may see:

kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable

In this case, it may be a good idea to supply an alternate clock-source to the kernel.

First, to check what clock-sources are available issue:

cat /sys/devices/system/clocksource/clocksource0/available_clocksource

which will return some of the following:

kvm-clock tsc hpet acpi_pm

To check the currently selected clock-source, issue:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

which will return the clock-source that Linux considers unstable.

In order to fix the unstable clock-source problem, edit /etc/default/grub or /etc/lilo.conf and add the following kernel parameter:

clocksource=acpi_pm

where acpi_pm is one of the available clock-sources. After which, a restart is required to pick-up the changes (remember to run lilo again if you are using lilo).

In order to clone a virtual machine with virsh, first connect and list all the machines as root:

virsh -c qemu:///system

then on the virsh console, issue:

list --all 

and shutdown the virtual machine that you wish to clone:

shutdown example.domain

You can then issue list –all again to check when the virtual machine has been shut down:

 -     example.domain                  shut off

Now exit the virsh command prompt with Ctrl+D and issue as root:

virt-clone --connect=qemu:///system -o example.domain -n new.domain -f /var/lib/libvirt/images/new.domain.img

where:

• example.domain represents the domain to clone
• new.domain represents the new domain to clone
• /var/lib/libvirt/images/new.domain.img is full path to the file where you want to store the new virtual machine.

Depending on how large the virtual machine is, after a while the new domain will be created. You should now go back to the virsh command line with:

virsh -c qemu:///system

and edit the new domain with:

edit new.domain

in order to make any last minute change to the VM.

After that, you can start the new domain / virtual machine using the virsh command line by issuing:

start new.domain

Alternatively, do not forget to set the autostart flag on the newly created virtual machine in case you want it to start automatically once the system boots:

autostart new.domain

Some resume states get corrupted which sometimes leads to the following error message when starting a domain:

error: failed to get domain 'testbed'
error: Unable to read from monitor: Connection reset by peer

In order to remove the saved state for the domain testbed issue the following command:

virsh managedsave-remove testbed

The boot-order can be changed by editing the domain name and including the boot-order devices:

<os>
  <type>hvm</type>
  <loader>/usr/lib/xen/boot/hvmloader</loader>
  <boot dev='network'/>
  <boot dev='cdrom'/>
  <boot dev='hd'/>
  <bootmenu enable='yes'/>
</os>

Virtual Machines can be imported using the –import parameter with virt-install. For example, the following command could be used:

virt-install -n NAME \    # the name of the domain to be created
    -r 2048 \             # the amount of RAM to be assigned to the virtual machine 
    --disk  /var/lib/libvirt/images/NAME.IMG,device=disk,bus=virtio \    # the disk image NAME.IMG to be imported using the virtio driver
    -w bridge=br0,model=virtio \                                         # bridge the network to br0 using the virtio driver
    --vnc \               # enable VNC
    --noautoconsole \     # do not open a console after importing
    --import              # and import

In case you have been accustomed to use the virtio bus, you will notice that booting off a Windows ISO and attempting to install will not find any disks connected. Switch the bus to ide and delete the address line (it will be recreated when you save the domain). You can then install Windows and install the virtio drivers and then switch back the bus to virtio.

First determine an available loop device by issuing:

losetup -f 

which will display a device such as /dev/loop0. Then attach the image file to the loop device:

lopsetup /dev/loop0 /var/lib/libvirt/images/guest.img

You can check that the partitions are there by using fdisk.

Next we run kpartx to create a map to the partitions available in the image file:

kpartx -av /dev/loop0

this will create device files under /dev/mapper/, for example, for two partitions on loop0, it will create /dev/mapper/loop0p1 and /dev/mapper/loop0p2.

These can then be mounted using mount, for example, to mount the second partition, you would issue:

mount /dev/mapper/loop0p2 /mnt/transfer

To enable the shutdown of Windows guests running under qemu/kvm, the following settings should be made:

• Using the group policy editor (run gpedit.msc) make sure that Computer Configuration→Windows Settings→Security Settings→Local Policies→Security Options→Shutdown: Allow system to be shut down without having to log on is set to Enabled.
• Using the registry editor (run regedit) make sure that the DWORD located at HKEY_LOCAL_MACHINE→SOFTWARE→Microsoft→Windows NT→CurrentVersion→Windows→ShutdownWarningDialogTimeout is set to anything else other than 0xffffffff (for example 1). In case the DWORD does not exist, create it and set it to a positive decimal.

You can now issue: shutdown windows.guest to shutdown a Windows domain named windows.guest.

Virtual machines (domains) are tied into the virsh system tightly such that there is no simple solution to renaming them except following the same steps:

• shut down the virtual machine with virsh shutdown.
• dump the domain you want to rename with virsh dumpxml DOMAIN > DOMAIN.xml - where DOMAIN is the virtual machine / domain you want to rename.
• edit the DOMAIN.xml file and locate the name tag and rename it appropriately.
• undefine the old domain with virsh undefine DOMAIN - where DOMAIN is the virtual machine / domain you want to rename.
• import the old domain with virsh define /path/to/DOMAIN.xml

You may want to actually save the new XML at /etc/libvirt/qemu/ where all the other domain XML files are located before importing the domain.

If you are running a Windows guest under libvirt using a virtio adapter type, you may notice that the Windows guest will not be able to acquire an IP address using DHCP.

To fix that TSO has to be disabled on the virtio interface. In order to do that, launch a registry editor and navigate to HKEY_LOCAL_MACHINE→SYSTEM→CurrentControlSet→Services→Tcpip→Parameters and add a new DWORD (32 bit). Give it the name DisableTaskOffload and set the value to 1. After that, restart the Windows guest and DHCP should work fine.

Guest machines can be configured to use the host random hardware for pseudo-random number generation. In order to do this you will either need egd or haveged to generate more entropy.

Using haveged you just need to install the daemon:

aptitude install haveged

and then add to the guest machine the configuration lines:

    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
    </rng> 

and restart the guest.

To use egd, you would have to install:

aptitude install egd

and then run egd in the background:

egd.pl --debug-client --nofork localhost:8888

where 8888 is the port to listen on.

After that, you would add the following configuration lines to the guest:

 <rng model='virtio'>
   <backend model='egd' type='tcp'>
     <source mode='connect' host='127.0.0.1' service='8888'/>
   </backend>
 </rng>

In order to test, you would log-in to a machine and verify with:

lsmod | grep virt

that the virtio_rng module is loaded.

Using losetup and partx images can be mounted from files. Let's say the image file is domain.img and we would like to detect and create a device for all partitions inside the image. We would first mount the image over loopback using losetup:

losetup -v -f /path/to/domain.img

This should create a loop device such as /dev/loop0 and display it. Next, we can list the partitions with partx:

partx --show /dev/loop0

To make partix add loop devices for all partitions on the image, we issue:

partx -v --add /dev/loop0

which should create devices such as /dev/loop0p1 (first partition), /dev/loop0p2 (second partition), etc…

To mount a shared folder inside the guest shared with the host, connect with virsh and edit the domain to add the following lines:

<filesystem type='mount' accessmode='mapped'>
    <source dir='/mnt/guest_shared'/>
    <target dir='host_share'/>
</filesystem>

This will expose the folder /mnt/guest_shared to the guest. The guest will then be able to mount the folder from the host /mnt/guest_shared by using the name host_share. Inside the guest, you would run:

mount -t 9p -o trans=virtio,version=9p2000.L host_share /mnt/shared/

which will mount the folder /mnt/guest_shared from the server to the folder /mnt/shared on the guest.

First, the guest needs to have a device defined for a CDROM. After that, the command:

virsh attach-disk domain /dev/sr0 hdc --type cdrom

or, for Windows:

virsh attach-disk domain /mnt/isos/cd.iso hdc --type cdrom

executed on the host operating system will mount the ISO inside the domain. The converse virsh command is detach-disk.

First, in order to make sure that libvirt will boot off the CD-ROM or Floppy and then proceed to the next bootable device, edit the domain and make sure that the lines:

    <boot dev='cdrom'/>
    <boot dev='hd'/>

are added inside the os tag.

Then, to eject a CD-ROM or Floppy, issue:

virsh change-media domain drive --eject --config

where:

• domain is the domain for which to eject the CD-ROM,
• drive is the bus device, ie hdc, hdd, et…,
• –eject will eject the CD-ROM,
• –config will update the domain XML to make the change persistent across reboots.

Conversely, to insert a CD-ROM or floppy, issue:

virsh change-media domain drive /var/lib/libvirt/iso/virtio-win.iso --insert --config

To make a virtual machine autostart on virsh start, issue on the virsh command line:

autostart domain

where domain is the name of the virtual machine.

In order to remove the autostart flag such that a virtual machine will not be start on virsh boot, issue:

autostart domain --disable

To enable Jumbo frames on a bridge, you will need to ensure that all member interfaces of the bridge are also set to use Jumbo frames.

Create an udev persistent network file (or edit it if it is available) on the machine hosting the virtual machines guests at /etc/udev/rules.d/70-persistent-net.rules with the following contents:

# LibVirt Jumbo Frames
SUBSYSTEM=="net", ACTION=="add", KERNEL=="vnet*", ATTR{mtu}="9000"
# Tap Interfaces Jumbo Frames
SUBSYSTEM=="net", ACTION=="add", KERNEL=="tap*", ATTR{mtu}="9000"

Edit /etc/network/interfaces and amend all configured interfaces to set the interface MTU to 9000. For instance, for an interface eth0 that will be a bridge member, you would have something like:

# Part of Bridge
allow-hotplug eth0
iface eth0 inet static
        # other options...
        mtu 9000

In case you are not on Sid or Etch, then you may need to add the line:

pre-up /sbin/ifconfig $IFACE mtu 1492

after the mtu 9000 line to make sure that dynamic interfaces also have the MTU set.

Additionally, in /etc/network/interfaces, make sure that the bridge also has the MTU set appropriately:

auto br0
iface br0 inet static
        # other options...
        mtu 9000
        pre-up /sbin/ifconfig $IFACE mtu 9000

Next, for all interfaces configured in /etc/sysconfig/network-scripts/ add the parameter:

MTU=9000

in order to enable Jumbo frames.

Similar to other virtualization software, qemu comes bundled with an agent that runs on the guest and facilitates various features. To install the qemu guest agent, first the agent itself has to be installed inside the guest, for instance, on Debian by issuing:

aptitude install qemu-guest-agent

and then the domain definition has to be changed to include:

    <channel type='unix'>
      <source mode='bind'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
    </channel>

inside the devices tags.

Shut down the domain, or destroy it if it cannot be shutdown:

virsh destroy mydomain.db

where:

• mydomain.db is the domain name.

Undefine the domain from libvirt:

virsh undefine mydomain.db

Finally, remove the volume from the pool:

virsh vol-delete --pool vms mydomain.db

where:

• vms is the libvirt storage pool for the domain mydomain.db.

Given a passed-through folder defined in the domain configuration file:

    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/mnt/cdrom'/>
      <target dir='cd'/>
    </filesystem>

virtual machines running under qemu / kvm have to be started as root. To accomplish that, edit /etc/default/libvirt/qemu.conf and set:

user = "root"
group = "root"

Additionally, the following setting:

clear_emulator_capabilities = 0

has to be added to /etc/default/libvirt/qemu.conf.

Once the virtual machine boots, the share can then be automatically mounted by adding the following line:

cd  /mnt/share     9p      defaults,trans=virtio,version=9p2000.L,posixacl 0       0

to /etc/fstab, where

• cd is the name of the share exposed by the host,
• /mnt/share is the path to a directory where the host share will be mounted,
• posixacl turns on passing POSIX ACLs (very convenient for using getfacl and setfacl to fine tune permissions).

libvirt allows using:

<target dev='...'/>

to set the custom name for bridge interfaces such as vnet0, vnet1, etc…

However, if the name of the interface contains the substring vnet, libvirt will ignore the entry and will not save it, for instance, setting the interface name with:

<target dev='vnet888'/>

will just be ignored by libvirt.

To resolve this issue, use a more custom-tailored interface name, for instance:

<target dev='mynet0'/>

without containing the vnet substring.