<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> Hi,<o:p></o:p> <o:p> </o:p> I’m doing a setup with Mellanox ConnectX-4 (MCX416A-CCA) NICs.<o:p></o:p> <o:p> </o:p> I’m using:<o:p></o:p> <o:p> </o:p> OS SLES 15 SP2<o:p></o:p> DPDK 19.11.4 (the official supported version for SLES 15 SP2)<o:p></o:p> MLNX_OFED_LINUX-5.5-1.0.3.2-sles15sp2-x86_64 (the latest one)<o:p></o:p> Mellanox adapters firmware 12.28.2006 (corresponding to this MLNX_OFED version)<o:p></o:p> kernel 5.3.18-24.34-default<o:p></o:p> <o:p> </o:p> <o:p> </o:p> This is my SRIOV configuration for DPDK capable PCI slots:<o:p></o:p> <o:p> </o:p> {<o:p></o:p> "resourceName": "mlnx_sriov_netdevice",<o:p></o:p> "resourcePrefix": "mellanox.com",<o:p></o:p> "isRdma": true,<o:p></o:p> "selectors": {<o:p></o:p> "vendors": ["15b3"],<o:p></o:p> "devices": ["1014"],<o:p></o:p> "drivers": ["mlx5_core"],<o:p></o:p> "pciAddresses": ["0000:d8:00.2", "0000:d8:00.3", "0000:d8:00.4", "0000:d8:00.5"],<o:p></o:p> "isRdma": true<o:p></o:p> }<o:p></o:p> <o:p> </o:p> The sriov device plugin starts without problems, the devices are correctly allocated:<o:p></o:p> <o:p> </o:p> {<o:p></o:p> "cpu": "92",<o:p></o:p> "ephemeral-storage": "419533922385",<o:p></o:p> "hugepages-1Gi": "8Gi",<o:p></o:p> "hugepages-2Mi": "4Gi",<o:p></o:p> "intel.com/intel_sriov_dpdk": "0",<o:p></o:p> "intel.com/sriov_cre": "3",<o:p></o:p> "mellanox.com/mlnx_sriov_netdevice": "4",<o:p></o:p> "mellanox.com/sriov_dp": "0",<o:p></o:p> "memory": "183870336Ki",<o:p></o:p> "pods": "110"<o:p></o:p> }<o:p></o:p> <o:p> </o:p> The Mellanox NICs are binded to the kernel driver mlx5_core:<o:p></o:p> <o:p> </o:p> pcgwpod009-c04:~ # dpdk-devbind --status<o:p></o:p> <o:p> </o:p> Network devices using kernel driver<o:p></o:p> ===================================<o:p></o:p> 0000:18:00.0 'Ethernet Controller 10G X550T 1563' if=em1 drv=ixgbe unused=vfio-pci<o:p></o:p> 0000:18:00.1 'Ethernet Controller 10G X550T 1563' if=em2 drv=ixgbe unused=vfio-pci<o:p></o:p> 0000:19:00.0 'Ethernet Controller 10G X550T 1563' if=em3 drv=ixgbe unused=vfio-pci<o:p></o:p> 0000:19:00.1 'Ethernet Controller 10G X550T 1563' if=em4 drv=ixgbe unused=vfio-pci<o:p></o:p> 0000:3b:00.0 'MT27700 Family [ConnectX-4] 1013' if=enp59s0f0 drv=mlx5_core unused=vfio-pci<o:p></o:p> 0000:3b:00.1 'MT27700 Family [ConnectX-4] 1013' if=enp59s0f1 drv=mlx5_core unused=vfio-pci<o:p></o:p> 0000:5e:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=p3p1 drv=ixgbe unused=vfio-pci<o:p></o:p> 0000:5e:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=p3p2 drv=ixgbe unused=vfio-pci<o:p></o:p> 0000:5e:10.0 '82599 Ethernet Controller Virtual Function 10ed' if= drv=ixgbevf unused=vfio-pci<o:p></o:p> 0000:5e:10.2 '82599 Ethernet Controller Virtual Function 10ed' if=p3p1_1 drv=ixgbevf unused=vfio-pci<o:p></o:p> 0000:5e:10.4 '82599 Ethernet Controller Virtual Function 10ed' if= drv=ixgbevf unused=vfio-pci<o:p></o:p> 0000:5e:10.6 '82599 Ethernet Controller Virtual Function 10ed' if=p3p1_3 drv=ixgbevf unused=vfio-pci<o:p></o:p> 0000:af:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=p4p1 drv=ixgbe unused=vfio-pci<o:p></o:p> 0000:af:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=p4p2 drv=ixgbe unused=vfio-pci<o:p></o:p> 0000:d8:00.0 'MT27700 Family [ConnectX-4] 1013' if=enp216s0f0 drv=mlx5_core unused=vfio-pci<o:p></o:p> 0000:d8:00.1 'MT27700 Family [ConnectX-4] 1013' if=enp216s0f1 drv=mlx5_core unused=vfio-pci<o:p></o:p> 0000:d8:00.2 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=enp216s0f2 drv=mlx5_core unused=vfio-pci<o:p></o:p> 0000:d8:00.3 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=enp216s0f3 drv=mlx5_core unused=vfio-pci<o:p></o:p> 0000:d8:00.4 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=enp216s0f4 drv=mlx5_core unused=vfio-pci<o:p></o:p> 0000:d8:00.5 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=enp216s0f5 drv=mlx5_core unused=vfio-pci<o:p></o:p> <o:p> </o:p> The interfaces are up:<o:p></o:p> <o:p> </o:p> pcgwpod009-c04:~ # ibdev2netdev -v<o:p></o:p> 0000:3b:00.0 mlx5_0 (MT4115 - MT1646K01301) CX416A - ConnectX-4 QSFP28 fw 12.28.2006 port 1 (ACTIVE) ==> enp59s0f0 (Up)<o:p></o:p> 0000:3b:00.1 mlx5_1 (MT4115 - MT1646K01301) CX416A - ConnectX-4 QSFP28 fw 12.28.2006 port 1 (ACTIVE) ==> enp59s0f1 (Up)<o:p></o:p> 0000:d8:00.0 mlx5_2 (MT4115 - MT1646K00538) CX416A - ConnectX-4 QSFP28 fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f0 (Up)<o:p></o:p> 0000:d8:00.1 mlx5_3 (MT4115 - MT1646K00538) CX416A - ConnectX-4 QSFP28 fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f1 (Up)<o:p></o:p> 0000:d8:00.2 mlx5_4 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f2 (Up)<o:p></o:p> 0000:d8:00.3 mlx5_5 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f3 (Up)<o:p></o:p> 0000:d8:00.4 mlx5_6 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f4 (Up)<o:p></o:p> 0000:d8:00.5 mlx5_7 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f5 (Up)<o:p></o:p> pcgwpod009-c04:~ #<o:p></o:p> <o:p> </o:p> <o:p> </o:p> But when I run my application the Mellanox adapters are probed and I obtain the following error:<o:p></o:p> <o:p> </o:p> {"proc_id":"6"},"message":"[pio] EAL: Probe PCI driver: mlx5_pci (15b3:1014) device: 0000:d8:00.4 (socket 1)"}<o:p></o:p> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] net_mlx5: unable to recognize master/representors on the multiple IB devices"}<o:p></o:p> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] common_mlx5: Failed to load driver = net_mlx5."}<o:p></o:p> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: Requested device 0000:d8:00.4 cannot be used"}<o:p></o:p> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: Bus (pci) probe failed."}<o:p></o:p> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] USER1: ports init fail in DPDK, expect 1 ports, actual 0 ports."}<o:p></o:p> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","severity":"error","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pktio_libpio_init] No network ports could be enabled!"}<o:p></o:p> <o:p> </o:p> Could you please help me with this issue?<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Thanks,<o:p></o:p> <o:p> </o:p> Roc�o<o:p></o:p> </div> </body> </html>