[dpdk-users] uio_pci_generic works but vfio-pci gives "Requested device xxx cannot be used" error

[McCullough, Harrison]

I am new to the world of drivers and low-level I/O, but I'm trying to learn how to use DPDK. I ran into this problem and was wondering if anybody has seen it before or knows what is going on.

I compiled the sample "hello world" application, and if I run it after binding the uio_pci_generic driver to a couple of interfaces it works fine. However, if I try running it after binding the vfio-pci driver to the same interfaces then I get this error:

EAL: Error - exiting with code: 1
  Cause: Requested device 0000:06:00.0 cannot be used

This is a transcript of the commands I was running. I also included the output of lspci in case that helps.

Thank you!

transcript:
$ ./tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:05:00.0 '82576 Gigabit Network Connection' if=p1p1 drv=igb unused=vfio-pci,uio_pci_generic *Active*
0000:05:00.1 '82576 Gigabit Network Connection' if=p1p2 drv=igb unused=vfio-pci,uio_pci_generic 
0000:06:00.0 '82576 Gigabit Network Connection' if=p1p3 drv=igb unused=vfio-pci,uio_pci_generic 
0000:06:00.1 '82576 Gigabit Network Connection' if=p1p4 drv=igb unused=vfio-pci,uio_pci_generic 

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>
$ sudo ./tools/dpdk-devbind.py --bind=vfio-pci 06:00.0
$ sudo ./tools/dpdk-devbind.py --bind=vfio-pci 06:00.1
$ ./tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:06:00.0 '82576 Gigabit Network Connection' drv=vfio-pci unused=uio_pci_generic
0000:06:00.1 '82576 Gigabit Network Connection' drv=vfio-pci unused=uio_pci_generic

Network devices using kernel driver
===================================
0000:05:00.0 '82576 Gigabit Network Connection' if=p1p1 drv=igb unused=vfio-pci,uio_pci_generic *Active*
0000:05:00.1 '82576 Gigabit Network Connection' if=p1p2 drv=igb unused=vfio-pci,uio_pci_generic 

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>
$ sudo ./examples/helloworld/build/helloworld -l 0-3
EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10e8 net_e1000_igb
EAL: PCI device 0000:05:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10e8 net_e1000_igb
EAL: PCI device 0000:06:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10e8 net_e1000_igb
EAL:   0000:06:00.0 failed to select IOMMU type
EAL: Error - exiting with code: 1
  Cause: Requested device 0000:06:00.0 cannot be used
$ sudo ./tools/dpdk-devbind.py --bind=uio_pci_generic 06:00.0
$ sudo ./tools/dpdk-devbind.py --bind=uio_pci_generic 06:00.1
$ ./tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:06:00.0 '82576 Gigabit Network Connection' drv=uio_pci_generic unused=vfio-pci
0000:06:00.1 '82576 Gigabit Network Connection' drv=uio_pci_generic unused=vfio-pci

Network devices using kernel driver
===================================
0000:05:00.0 '82576 Gigabit Network Connection' if=p1p1 drv=igb unused=vfio-pci,uio_pci_generic *Active*
0000:05:00.1 '82576 Gigabit Network Connection' if=p1p2 drv=igb unused=vfio-pci,uio_pci_generic 

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>
$ sudo ./examples/helloworld/build/helloworld -l 0-3
EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10e8 net_e1000_igb
EAL: PCI device 0000:05:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10e8 net_e1000_igb
EAL: PCI device 0000:06:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10e8 net_e1000_igb
EAL: PCI device 0000:06:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10e8 net_e1000_igb
hello from core 1
hello from core 2
hello from core 3
hello from core 0
$ lspci
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
03:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
04:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
04:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
fe:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05)
fe:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05)
fe:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05)
fe:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05)
fe:02.4 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 1 (rev 05)
fe:02.5 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 1 (rev 05)
fe:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05)
fe:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05)
fe:03.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller RAS Registers (rev 05)
fe:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05)
fe:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05)
fe:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05)
fe:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05)
fe:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05)
fe:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05)
fe:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05)
fe:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05)
fe:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05)
fe:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05)
fe:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05)
fe:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05)
fe:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05)
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.2.0-42-generic root=/dev/mapper/mtznjrsv228--vg-root ro default_hugepagesz=1G hugepagesz=1G hugepages=4 iommu=pt intel_iommu=on