[dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3problem!

Olga Shern olgas at mellanox.com
Mon Apr 27 10:50:26 CEST 2015


Hi Kang,

I am glad you were able to fix the problem.
But I think there is some confusion here, it means that we need have better documentation ☺
First step to work with Mellanox NIC is to install proper MLNX_OFED and FW from Mellanox.com
Seems that you did it.
Then you need to run /etc/init.d/openibd restart to load the needed modules.

For DPDK,  you need to configure your card to work as Ethernet, Mellanox VPI cards can be either Infiniband or Ethernet.
Use  connectx_port_config to configure it, as you did
After that you can run ibdev2netdev and ifconfig and verify that you have interfaces for Mellanox NIC ports.

Then you can compile DPDK with Mellanox PMD enabled. Mellanox PMD can be dynamically linked with the MLNX_OFED installed libibverbs and libmlx4 – but the performance you get is not the best one.
Mellanox PMD can also be statically compiled with optimized libibverbs and libmlx4 that can be downloaded from dpdk.org site.

Let me know if it clear.
Best Regards,
Olga

From: Arthas [mailto:kangzy1982 at qq.com]
Sent: Monday, April 27, 2015 6:07 AM
To: Arthas; Olga Shern; dev
Subject: Re: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3problem!

Hi, Olga,
I think I have solved this problem.
libibverbs & libmlx4 can't use dpdp.org and can't compile with static linking.
I try use libibverbs-1.1.8mlnx1-OFED.2.4.45.ga305acd.src.rpm & libmlx4-1.0.6mlnx1-OFED.2.4.0.1.2.src and compile with dynamic. This two package in MLNX_OFED version 2.4-1.0.0 source package.
and I use ofed_script  connectx_port_config  config Card with Ethernet mode .
The testpmd was started with no error! :)

Thanks a million!! :)

Best Regards!
Kang

------------------ Original ------------------
From:  "Arthas";<kangzy1982 at qq.com<mailto:kangzy1982 at qq.com>>;
Date:  Mon, Apr 27, 2015 10:37 AM
To:  "Olga Shern"<olgas at mellanox.com<mailto:olgas at mellanox.com>>; "dev"<dev at dpdk.org<mailto:dev at dpdk.org>>;
Subject:  Re: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3problem!

3KS, Olga,
  My MLNX_OFED version 2.4-1.0.0 and libibverbs & libmlx4 download from dpdk.org.
 when i try to run ibdev2netdev , it don't output any message! :(
 Here is my environment.
 # uname -r
3.10.0-mlx
 # lsmod
mlx4_en 92824 0 - Live 0xffffffffa010f000 (O)
mlx4_ib 135033 0 - Live 0xffffffffa00e3000 (O)
ib_sa 24721 1 mlx4_ib, Live 0xffffffffa00d7000 (O)
ib_mad 31664 2 mlx4_ib,ib_sa, Live 0xffffffffa00ca000 (O)
mlx4_core 226516 2 mlx4_en,mlx4_ib, Live 0xffffffffa007c000 (O)
ib_uverbs 43309 0 - Live 0xffffffffa003f000 (O)
ib_core 79534 4 mlx4_ib,ib_sa,ib_mad,ib_uverbs, Live 0xffffffffa001d000 (O)
ib_addr 4273 2 ib_uverbs,ib_core, Live 0xffffffffa0017000 (O)
compat 4948 8 mlx4_en,mlx4_ib,ib_sa,ib_mad,mlx4_core,ib_uverbs,ib_core,ib_addr, Live 0xffffffffa0011000 (O)
 # mstflint  -d 02:00.0 q
Image type:      FS2
FW Version:      2.33.5100
FW Release Date: 25.1.2015
Product Version: 02.33.51.00
Rom Info:        type=PXE version=3.4.460 devid=4099
Device ID:       4099
Description:     Node             Port1            Port2            Sys image
GUIDs:           0002c90300faa480 0002c90300faa481 0002c90300faa482 0002c90300faa483
MACs:                                 0002c9faa481     0002c9faa482
VSD:
PSID:            MT_1090110019
# ibv_devices
libibverbs: Warning: dlopen(NULL) failed, assuming static linking.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
        When linking libibverbs statically, driver must be statically linked too.
    device                 node GUID
    ------              ----------------
 # ibv_devinfo  -l
libibverbs: Warning: dlopen(NULL) failed, assuming static linking.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
        When linking libibverbs statically, driver must be statically linked too.
0 HCAs found:

/sys/class/infiniband_verbs # ls
abi_version  uverbs0
 # find . -name "mlx4*"
./proc/irq/148/mlx4-32 at 0000:02:00.0<mailto:./proc/irq/148/mlx4-32 at 0000:02:00.0>
./proc/irq/147/mlx4-31 at 0000:02:00.0<mailto:./proc/irq/147/mlx4-31 at 0000:02:00.0>
./proc/irq/146/mlx4-30 at 0000:02:00.0<mailto:./proc/irq/146/mlx4-30 at 0000:02:00.0>
./proc/irq/145/mlx4-29 at 0000:02:00.0<mailto:./proc/irq/145/mlx4-29 at 0000:02:00.0>
./proc/irq/144/mlx4-28 at 0000:02:00.0<mailto:./proc/irq/144/mlx4-28 at 0000:02:00.0>
./proc/irq/143/mlx4-27 at 0000:02:00.0<mailto:./proc/irq/143/mlx4-27 at 0000:02:00.0>
./proc/irq/142/mlx4-26 at 0000:02:00.0<mailto:./proc/irq/142/mlx4-26 at 0000:02:00.0>
./proc/irq/141/mlx4-25 at 0000:02:00.0<mailto:./proc/irq/141/mlx4-25 at 0000:02:00.0>
./proc/irq/140/mlx4-24 at 0000:02:00.0<mailto:./proc/irq/140/mlx4-24 at 0000:02:00.0>
./proc/irq/139/mlx4-23 at 0000:02:00.0<mailto:./proc/irq/139/mlx4-23 at 0000:02:00.0>
./proc/irq/138/mlx4-22 at 0000:02:00.0<mailto:./proc/irq/138/mlx4-22 at 0000:02:00.0>
./proc/irq/137/mlx4-21 at 0000:02:00.0<mailto:./proc/irq/137/mlx4-21 at 0000:02:00.0>
./proc/irq/136/mlx4-20 at 0000:02:00.0<mailto:./proc/irq/136/mlx4-20 at 0000:02:00.0>
./proc/irq/135/mlx4-19 at 0000:02:00.0<mailto:./proc/irq/135/mlx4-19 at 0000:02:00.0>
./proc/irq/134/mlx4-18 at 0000:02:00.0<mailto:./proc/irq/134/mlx4-18 at 0000:02:00.0>
./proc/irq/133/mlx4-17 at 0000:02:00.0<mailto:./proc/irq/133/mlx4-17 at 0000:02:00.0>
./proc/irq/132/mlx4-16 at 0000:02:00.0<mailto:./proc/irq/132/mlx4-16 at 0000:02:00.0>
./proc/irq/131/mlx4-15 at 0000:02:00.0<mailto:./proc/irq/131/mlx4-15 at 0000:02:00.0>
./proc/irq/130/mlx4-14 at 0000:02:00.0<mailto:./proc/irq/130/mlx4-14 at 0000:02:00.0>
./proc/irq/129/mlx4-13 at 0000:02:00.0<mailto:./proc/irq/129/mlx4-13 at 0000:02:00.0>
./proc/irq/128/mlx4-12 at 0000:02:00.0<mailto:./proc/irq/128/mlx4-12 at 0000:02:00.0>
./proc/irq/127/mlx4-11 at 0000:02:00.0<mailto:./proc/irq/127/mlx4-11 at 0000:02:00.0>
./proc/irq/126/mlx4-10 at 0000:02:00.0<mailto:./proc/irq/126/mlx4-10 at 0000:02:00.0>
./proc/irq/125/mlx4-9 at 0000:02:00.0<mailto:./proc/irq/125/mlx4-9 at 0000:02:00.0>
./proc/irq/124/mlx4-8 at 0000:02:00.0<mailto:./proc/irq/124/mlx4-8 at 0000:02:00.0>
./proc/irq/123/mlx4-7 at 0000:02:00.0<mailto:./proc/irq/123/mlx4-7 at 0000:02:00.0>
./proc/irq/122/mlx4-6 at 0000:02:00.0<mailto:./proc/irq/122/mlx4-6 at 0000:02:00.0>
./proc/irq/121/mlx4-5 at 0000:02:00.0<mailto:./proc/irq/121/mlx4-5 at 0000:02:00.0>
./proc/irq/120/mlx4-4 at 0000:02:00.0<mailto:./proc/irq/120/mlx4-4 at 0000:02:00.0>
./proc/irq/119/mlx4-3 at 0000:02:00.0<mailto:./proc/irq/119/mlx4-3 at 0000:02:00.0>
./proc/irq/118/mlx4-2 at 0000:02:00.0<mailto:./proc/irq/118/mlx4-2 at 0000:02:00.0>
./proc/irq/117/mlx4-1 at 0000:02:00.0<mailto:./proc/irq/117/mlx4-1 at 0000:02:00.0>
./proc/irq/116/mlx4-async at pci:0000:02:00.0<mailto:./proc/irq/116/mlx4-async at pci:0000:02:00.0>
./proc/driver/mlx4_ib
./sys/bus/pci/drivers/mlx4_core
./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port1_mtu
./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port2_mtu
./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/mlx4_0
./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port1
./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port2
./sys/class/infiniband/mlx4_0
./sys/module/ib_sa/holders/mlx4_ib
./sys/module/compat/holders/mlx4_en
./sys/module/compat/holders/mlx4_ib
./sys/module/compat/holders/mlx4_core
./sys/module/mlx4_en
./sys/module/mlx4_ib
./sys/module/ib_mad/holders/mlx4_ib
./sys/module/ib_core/holders/mlx4_ib
./sys/module/mlx4_core
./sys/module/mlx4_core/holders/mlx4_en
./sys/module/mlx4_core/holders/mlx4_ib
./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx4
./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/infiniband/hw/mlx4
./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko
./sdd/tmp/mlx4.driver
./sdd/tmp/mlx4
./sdd/tmp/mlx4/mlx4.hw
./sdd/tmp/mlx4/mlx4.config
./sdd/tmp/mlx4/mlx4.img.2.10.700
./sdd/tmp/etc/modprobe.d/mlx4.conf
./etc/libibverbs.d/mlx4.driver
./etc/modprobe.d/mlx4.conf




 ------------------ Original ------------------
  From:  "Olga Shern";<olgas at mellanox.com<mailto:olgas at mellanox.com>>;
 Date:  Sun, Apr 26, 2015 03:28 PM
 To:  "Arthas"<kangzy1982 at qq.com<mailto:kangzy1982 at qq.com>>; "dev"<dev at dpdk.org<mailto:dev at dpdk.org>>;

 Subject:  RE: RE: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3problem!




Hi Kang,



Sorry for the late response.

ConnectX-3 VPI card that you have is supported. So there shouldn’t be any problem.

Please make sure to follow the instructions in http://www.dpdk.org/doc/guides/nics/mlx4.html

What MLNX_OFED did you install?

What is the output of ibdev2netdev?

Make sure, mlx4_core parameter log_num_mgm_entry_size = -1



Best Regards,

Olga





From: Arthas [mailto:kangzy1982 at qq.com]
Sent: Wednesday, April 22, 2015 6:56 AM
To: Olga Shern; dev
Subject: Re: RE: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3 problem!



 3KS, Olga,


    I try you suggestion and it can't start yet. My Mellanox ConnectX-3 card is "ConnectX®-3 VPI adapter card, dual-port QSFP, FDR IB   (56Gb/s) and 40/56GbE, PCIe3.0 x8 8GT/s, tall bracket,RoHS R6". May be the problem?





 # mstflint  -d 02:00.0 q
Image type:      FS2
FW Version:      2.33.5100
FW Release Date: 25.1.2015
Product Version: 02.33.51.00
Rom Info:        type=PXE version=3.4.460 devid=4099
Device ID:       4099
Description:     Node             Port1            Port2            Sys image
GUIDs:           0002c90300faa480 0002c90300faa481 0002c90300faa482 0002c90300faa483
MACs:                                 0002c9faa481     0002c9faa482
VSD:
PSID:            MT_1090110019



Best Regards,


Kang


------------------ Original ------------------


From:  "Olga Shern";<olgas at mellanox.com<mailto:olgas at mellanox.com>>;


Date:  Tue, Apr 21, 2015 09:34 PM


To:  "Arthas"<kangzy1982 at qq.com<mailto:kangzy1982 at qq.com>>; "dev"<dev at dpdk.org<mailto:dev at dpdk.org>>;


Subject:  RE: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3 problem!






Hi Kang,

You probably compiled  the downloaded libraries using dynamic linkage, prior to the static one you specified here.
You need to remove  them from /usr/local/libs and also please delete /usr/local/include/infiniband

Let me know if it solves your problem.

Best Regards,
Olga


-----Original Message-----
From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of ZhanYing Kang
Sent: Tuesday, April 21, 2015 11:45 AM
To: dev
Subject: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3 problem!

When  I'm  run testpmd with DPDK 2.0 and the Mellanox ConnectX-3. I get a error.

Here are my questions:
1> lspci |grep Mell
02:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

2> compile libibverbs-1.1.7mlnx1 && libmlx4-1.0.5mlnx1 export EXTRA_CFLAGS=-I/DPDK/mlx4/install/usr/local/include
export EXTRA_LDFLAGS=-L/DPDK/mlx4/install/usr/local/lib

3> compile dpdk v2.0 with x86_64-native-linuxapp-gcc config and
3> CONFIG_RTE_LIBRTE_MLX4_PMD=y

4> modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib  lsmod mlx4_ib 135033 0 - Live 0xffffffffa0215000 (O) ib_sa 24721 1 mlx4_ib, Live 0xffffffffa0209000 (O) ib_mad 31664 2 mlx4_ib,ib_sa, Live 0xffffffffa01fc000 (O) mlx4_en 92808 0 - Live 0xffffffffa01b1000 (O) mlx4_core 226516 2 mlx4_ib,mlx4_en, Live 0xffffffffa0163000 (O) ib_uverbs 43309 0 - Live 0xffffffffa0133000 (O) ib_core 79534 4 mlx4_ib,ib_sa,ib_mad,ib_uverbs, Live 0xffffffffa0100000 (O) ib_addr 4273 2 ib_uverbs,ib_core, Live 0xffffffffa00fa000 (O) compat 4948 8 mlx4_ib,ib_sa,ib_mad,mlx4_en,mlx4_core,ib_uverbs,ib_core,ib_addr, Live 0xffffffffa00ad000 (O)  5> run testpmd
  # ./testpmd-mlx  -c 0xff00 -n 4 -w 0000:02:00.0 -- --rxq=2 --txq=2 -i
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 6 on socket 0
EAL: Detected lcore 7 as core 7 on socket 0
EAL: Detected lcore 8 as core 0 on socket 0
EAL: Detected lcore 9 as core 1 on socket 0
EAL: Detected lcore 10 as core 2 on socket 0
EAL: Detected lcore 11 as core 3 on socket 0
EAL: Detected lcore 12 as core 4 on socket 0
EAL: Detected lcore 13 as core 5 on socket 0
EAL: Detected lcore 14 as core 6 on socket 0
EAL: Detected lcore 15 as core 7 on socket 0
EAL: Support maximum 32 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: Setting up memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f5a2b800000 (size = 0x200000)
EAL: Ask a virtual area of 0x31000000 bytes
EAL: Virtual area found at 0x7f59fa600000 (size = 0x31000000)
EAL: Ask a virtual area of 0x7ec00000 bytes
EAL: Virtual area found at 0x7f597b800000 (size = 0x7ec00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f597b400000 (size = 0x200000)
EAL: Ask a virtual area of 0x250000000 bytes
EAL: Virtual area found at 0x7f572b200000 (size = 0x250000000)
EAL: Requesting 6144 pages of size 2MB from socket 0
EAL: TSC frequency is ~2394227 KHz
EAL: Master lcore 8 is ready (tid=2e1b7900;cpuset=[8])
EAL: lcore 11 is ready (tid=299b7700;cpuset=[11])
EAL: lcore 14 is ready (tid=281b4700;cpuset=[14])
EAL: lcore 15 is ready (tid=279b3700;cpuset=[15])
EAL: lcore 12 is ready (tid=291b6700;cpuset=[12])
EAL: lcore 9 is ready (tid=2caaf700;cpuset=[9])
EAL: lcore 10 is ready (tid=2c2ae700;cpuset=[10])
EAL: lcore 13 is ready (tid=289b5700;cpuset=[13])
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF: false)
PMD: librte_pmd_mlx4: 2 port(s) detected
PMD: librte_pmd_mlx4: bad state for port 1: "down" (1)
PMD: librte_pmd_mlx4: port 1 MAC address is 02:02:c9:fa:a4:81
PMD: librte_pmd_mlx4: bad state for port 2: "down" (1)
PMD: librte_pmd_mlx4: port 2 MAC address is 00:02:c9:fa:a4:82 Interactive-mode selected Configuring Port 0 (socket 0)
PMD: librte_pmd_mlx4: 0x7dffe0: TX queues number update: 0 -> 2
PMD: librte_pmd_mlx4: 0x7dffe0: RX queues number update: 0 -> 2
PMD: librte_pmd_mlx4: 0x7fffc81e67e0: flow configuration failed, errno=38: Function not implemented
PMD: librte_pmd_mlx4: 0x7fffc81e67e0: flow configuration failed, errno=38: Function not implemented
PMD: librte_pmd_mlx4: 0x7dffe0: QP flow attachment failed: Function not implemented Fail to configure port 0
EAL: Error - exiting with code: 1
  Cause: Start ports failed

Best,
Kang!


More information about the dev mailing list