[dpdk-users] Cannot attach DPDK NIC port to OVS

[Vincenzo Maffione]

vfio-pci is loaded and bound to the NIC, as shown by dpdk-devbind.

$ sudo tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=vfio-pci
unused=ixgbe

Network devices using kernel driver
===================================
0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e
unused=vfio-pci
0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp1s0f1
drv=ixgbe unused=vfio-pci
0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
if=enp6s0 drv=r8169 unused=vfio-pci
0000:07:00.0 '82574L Gigabit Network Connection' if=enp7s0 drv=e1000e
unused=vfio-pci

[...]

And I get:

Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: PCI device 0000:01:00.0
on NUMA socket -1
Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL:   probe driver:
8086:10fb net_ixgbe
Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Requested device
0000:01:00.0 cannot be used
Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Driver cannot attach
the device (0000:01:00.0)
Apr 20 18:19:22 arch-server ovs-vswitchd[650]:
2017-04-20T16:19:22Z|00027|netdev_dpdk|WARN|Error attaching device
'0000:01:00.0' to DPDK
Apr 20 18:19:22 arch-server ovs-vswitchd[650]:
2017-04-20T16:19:22Z|00028|netdev|WARN|dpdk-p1: could not set configuration
(Invalid argumen
Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: PCI device 0000:01:00.0
on NUMA socket -1
Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL:   probe driver:
8086:10fb net_ixgbe
Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Requested device
0000:01:00.0 cannot be used
Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Driver cannot attach
the device (0000:01:00.0)
Apr 20 18:19:22 arch-server ovs-vswitchd[650]:
ovs|00027|netdev_dpdk|WARN|Error attaching device '0000:01:00.0' to DPDK
Apr 20 18:19:22 arch-server ovs-vswitchd[650]:
ovs|00028|netdev|WARN|dpdk-p1: could not set configuration (Invalid
argument)


Thanks,
  Vincenzo


2017-04-20 18:13 GMT+02:00 Shivaram Mysore <shivaram.mysore at gmail.com>:

> I have a feeling that vfio-pci is not installed.  Note that you will have
> to edit /etc/modules to load on boot.
>
> You can also check by
> # lsmod | grep vfio
>
> To load on demand - use
> # modprobe vfio-pci
>
>
>
> On Thu, Apr 20, 2017 at 9:03 AM, Vincenzo Maffione <v.maffione at gmail.com>
> wrote:
>
>> Hi,
>>   I installed OVS 2.7.0 and use uio_pci_generic in place of vfio-pci, and
>> the error disappeared.
>>
>> Thanks,
>>   Vincenzo
>>
>> 2017-04-15 17:56 GMT+02:00 Shivaram Mysore <shivaram.mysore at gmail.com>:
>>
>>> You should look at emails from -(Hint:  search DPDK in subject)
>>>
>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Marc
>>> h/thread.html
>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Apri
>>> l/thread.html
>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Apri
>>> l/044155.html
>>>
>>> If possible, I would recommend using Ubuntu 17.04 with OVS 2.7 with DPDK
>>> packages from http://packages.wand.net.nz/ which can simplify your
>>> deployment.
>>>
>>> /Shivaram
>>>
>>> On Fri, Apr 14, 2017 at 4:04 AM, Vincenzo Maffione <
>>> vincenzo.maffione at ing.unipi.it> wrote:
>>>
>>>> Hi,
>>>>   I would like to attach one of my ixgbe NIC ports to an OpenVSwitch
>>>> instance, but I keep getting an unexpected error, see below.
>>>>
>>>> I've successfully bound both my ixgbe NIC ports to DPDK, using VFIO:
>>>>
>>>> # tools/dpdk-devbind.py --status
>>>>
>>>> Network devices using DPDK-compatible driver
>>>> ============================================
>>>> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
>>>> drv=vfio-pci
>>>> unused=ixgbe
>>>> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
>>>> drv=vfio-pci
>>>> unused=ixgbe
>>>>
>>>> Network devices using kernel driver
>>>> ===================================
>>>> 0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e
>>>> unused=vfio-pci
>>>> 0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
>>>> if=enp6s0 drv=r8169 unused=vfio-pci
>>>> 0000:07:00.0 '82574L Gigabit Network Connection' if=enp7s0 drv=e1000e
>>>> unused=vfio-pci
>>>>
>>>> Other network devices
>>>> =====================
>>>> <none>
>>>>
>>>> Crypto devices using DPDK-compatible driver
>>>> ===========================================
>>>> <none>
>>>>
>>>> Crypto devices using kernel driver
>>>> ==================================
>>>> <none>
>>>>
>>>> Other crypto devices
>>>> ====================
>>>> <none>
>>>>
>>>>
>>>> Then I created a bridge
>>>>
>>>> # ovs-vsctl add-br obr1 -- set bridge obr1 datapath_type=netdev
>>>>
>>>> But then when trying to add the dpdk NIC:
>>>>
>>>> # ovs-vsctl add-port obr1 dpdk-p0 -- set Interface dpdk-p0 type=dpdk
>>>> options:dpdk-devargs=0000:01:00.0
>>>>
>>>> the command fails, and I get this in the log:
>>>>
>>>> ovs-vswitchd[353]: 2017-04-14T10:54:06Z|00026|bridge|WARN|could not
>>>> open
>>>> network device dpdk-p0 (No such device)
>>>> ovs-vswitchd[353]: ovs|00026|bridge|WARN|could not open network device
>>>> dpdk-p0 (No such device)
>>>>
>>>> Note that I've successfully managed to setup an OVS instance with two
>>>> DPDK
>>>> vhost-user ports (each one assigned to a QEMU VM): this is why I'm
>>>> confident my OVS is built and configured correctly to work with DPDK.
>>>> Also,
>>>> DPDK by itself is working as expected, I can use dpdk pktgen.
>>>> Moreover, OVS log shows that DPDK is recognized
>>>>
>>>> [...]
>>>> ovs|00005|dpdk|INFO|DPDK Enabled, initializing
>>>> [...]
>>>>
>>>> Note that I'm carefully following the documentation here
>>>> http://docs.openvswitch.org/en/latest/intro/install/dpdk/ and here
>>>> http://docs.openvswitch.org/en/latest/howto/dpdk/
>>>>
>>>> More info:
>>>>
>>>> DPDK version: 16.11
>>>> OVS version: 2.6.1, configured with --with-dpdk
>>>> Host: Archlinux
>>>> /proc/cmdline: BOOT_IMAGE=../vmlinuz-linux-vfio
>>>> root=UUID=1c777330-3f00-4ab8-8d5d-d4b2b5d860b0 rw intel_iommu=on
>>>> hugepages=2048 pci=assign-busses pcie_acs_override=downstream
>>>> initrd=../initramfs-linux-vfio.img
>>>>
>>>> Can anybody help? I'm at a loss here.
>>>>
>>>> Thanks,
>>>>   Vincenzo
>>>>
>>>
>>>
>>
>>
>> --
>> Vincenzo Maffione
>>
>
>