fluxcoil wiki

### 1
# Run an intel system with "intel_iommu=on iommu=pt".
# The latter option is optional on older RHEL versions (i.e. 6),
# disables checks for better performance.
$ cat /proc/cmdline 
[..] intel_iommu=on iommu=pt

### 2
# verify that cgroups are created (only applicable to RHEL7)
$ find /sys/kernel/iommu_groups/ -type l

### 3
# inspect the systems pci devices
$ lspci -nvv | less
# For example, I want to export 03:00.1
03:00.1 0200: 14e4:165f
        Subsystem: 1028:0639
[..]
        Kernel driver in use: tg3
        Kernel modules: tg3
# => we want to pass through here device 03:00.1

### 4
# find out the devices in the same cgroup
$ ll /sys/bus/pci/devices/0000\:03\:00.1/iommu_group/devices/
total 0
[..] 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.3/0000:03:00.0
[..] 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.3/0000:03:00.1
# or via virsh:
$ virsh nodedev-list --tree
computer
  |
  [..]
  +- pci_0000_00_1c_3
  |   |
  |   +- pci_0000_03_00_0
  |   |   |
  |   |   +- net_em3_18_66_da_5a_ee_95
  |   |     
  |   +- pci_0000_03_00_1
# => So we will need to detach these 2 devices,
     pci_0000_03_00_0 and pci_0000_03_00_1, from the host

### 5
# get the virsh name for the device
$ virsh nodedev-list --cap pci
[..]
pci_0000_03_00_1
[..]

### 6
# detach the devices seen in 4)
$ virsh nodedev-dettach pci_0000_03_00_0
Device pci_0000_03_00_0 detached
$ virsh nodedev-dettach pci_0000_03_00_1
Device pci_0000_03_00_1 detached

### 5
# start the VM with the device passthrough
$ virt-install --name=guest1-rhel6-64 \
    --disk path=/var/lib/libvirt/images/guest1-rhel6-64.img,size=8 
    --vcpus=2 --ram=2048 \
    --location=http://ip/6.8/Server/x86_64/os/ 
    --network bridge=virbr0,model=virtio \
    --os-type=linux --os-variant=rhel6 \
    --host-device=pci_0000_03_00_1

$ virt-install [..] --host-device=pci_0000_03_00_1
Starting install...
Retrieving file vmlinuz...                                                                                                                               | 4.1 MB  00:00:10     
Retrieving file initrd.img...                                                                                                                            |  39 MB  00:00:12     
Allocating 'guest1-rhel6-64.img'                                                                                                                         | 8.0 GB  00:00:00     
ERROR    internal error: qemu unexpectedly closed the monitor: 2017-05-11T08:25:06.847580Z qemu-kvm: -device vfio-pci,host=03:00.1,id=hostdev0,bus=pci.0,addr=0x8: vfio: error, group 18 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2017-05-11T08:25:06.847605Z qemu-kvm: -device vfio-pci,host=03:00.1,id=hostdev0,bus=pci.0,addr=0x8: vfio: failed to get group 18
2017-05-11T08:25:06.847629Z qemu-kvm: -device vfio-pci,host=03:00.1,id=hostdev0,bus=pci.0,addr=0x8: Device initialization failed.
2017-05-11T08:25:06.847641Z qemu-kvm: -device vfio-pci,host=03:00.1,id=hostdev0,bus=pci.0,addr=0x8: Device 'vfio-pci' could not be initialized
Removing disk 'guest1-rhel6-64.img'                                                                                                                      |    0 B  00:00:00     
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start guest1-rhel6-64
otherwise, please restart your installation.

Although just this device is intended to be reached through, all devices in the group have to be in the cgroup, and to be detached from the host. Perform steps 4 and 6 from above.