<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hello,<o:p></o:p> <o:p> </o:p> Please confirm your NIC is configured to ETH and not IB, but running this command, for example:<o:p></o:p> mlxconfig -d /dev/mst/mt4119_pciconf0 q | grep LINK<o:p></o:p> <o:p> </o:p> You should see something like this:<o:p></o:p> LINK_TYPE_P1 ETH(2) LINK_TYPE_P2 ETH(2)<o:p></o:p> <o:p> </o:p> If not, please refer to this guide that explains how to configure the NIC as ETH:<o:p></o:p> <a href="https://support.mellanox.com/s/article/getting-started-with-connectx-4-100gb-s-adapter-for-linux">https://support.mellanox.com/s/article/getting-started-with-connectx-4-100gb-s-adapter-for-linux</a><o:p></o:p> <o:p> </o:p> Regards,<o:p></o:p> Asaf Penso<o:p></o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: fwefew 4t4tg <7532yahoo@gmail.com> Sent: Friday, April 8, 2022 3:08 AM To: users@dpdk.org Subject: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] not receiving (transmitting) packets<o:p></o:p> </div> </div> <o:p> </o:p> <div> I have two identical metal boxes running Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz. There is no virtualization; so I did NOT enable IOMMU. Both boxes are equipped with two Mellanox Technologies MT27710 NICs and two Intel NICs. The Intel NICs are out of scope. I am using the NIC at 01:00.1 for DPDK: # lspci | grep Eth 01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] 01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] 03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)<o:p></o:p> <div> <o:p> </o:p> </div> <div> ibv and mst utilities see both Mellanox NICs. DPDK's testpmd application sees the NIC and reports decent values except perhaps ibv_devinfo which reports Infiniband transport. My application sends UDP packets from one machine to another. I know this code works on AWS ENA NICs. The build for Mellaonx finds the NIC just like DPDK's testpmd reporting the same EAL info. There are no errors transmitting packets; all numbers in stats are exactly what they should be. However, the receiving side never sees any packets. It reports no errors and does not see any packets. Every call to rx_burst sees 0 packets. There is no firewall issue. I can use ncat in tcp and udp mode to send files between the machines just fine.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> My application is using CRC checksum offload for RX and TX (value 6). And I triple checked the MAC and IP addresses I use in the code. I believe they are fine. I set auto-negotiate on both the RX and TX side. I have also tried to set 10Gbs link speed. No help. I also ran testdpdk then tried to send it packets with ncat; testpmd never sees any packets either.<o:p></o:p> </div> <div> I have read through <a href="https://doc.dpdk.org/guides/nics/mlx5.html">https://doc.dpdk.org/guides/nics/mlx5.html</a> for proper setup. I found the following deviations: - <a href="https://doc.dpdk.org/guides/platform/mlx5.html#mlx5-common-env">https://doc.dpdk.org/guides/platform/mlx5.html#mlx5-common-env</a> says to set the link type to Eth: mlxconfig -d <mst device> query | grep LINK_TYPE. These devices do NOT have a link type and do not allow setting it. I am assuming it only works in Eth mode. - the latest and greatest MST toolkit wget <a href="https://www.mellanox.com/downloads/MFT/mft-4.18.0-106-x86_64-deb.tgz"> https://www.mellanox.com/downloads/MFT/mft-4.18.0-106-x86_64-deb.tgz</a> does not include the utiliity mlxdevm and neither does the OEFD install wget <a href="https://www.mellanox.com/downloads/ofed/MLNX_EN-5.5-1.0.3.2/mlnx-en-5.5-1.0.3.2-ubuntu20.04-x86_64.iso"> https://www.mellanox.com/downloads/ofed/MLNX_EN-5.5-1.0.3.2/mlnx-en-5.5-1.0.3.2-ubuntu20.04-x86_64.iso</a> include it. The ISO file does NOT have a utility called mlnxofedinstall. So I am not sure if I am missing something. The doc reads:<o:p></o:p> </div> The firmware, the libraries libibverbs, libmlx5, and mlnx-ofed-kernel modules are packaged in Mellanox OFED. After downloading, it can be installed with this command:<o:p></o:p> <div> ./mlnxofedinstall --dpdk So I just ran ./install --dpdk since the ISO image does have an install script taking the --dpdk argument.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> The doc <a href="https://doc.dpdk.org/guides/platform/mlx5.html#mlx5-common-env">https://doc.dpdk.org/guides/platform/mlx5.html#mlx5-common-env</a> mentions a whole bunch of instructions about SRIOV, SF ports etc but which requires mlxdevm. As I say above there is no such utility.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> So I am stuck. DETAILS:<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> uname -a Linux client 5.13.0-28-generic #31~20.04.1-Ubuntu SMP Wed Jan 19 14:08:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux root@client:~/Dev/reinvent/scripts# ibv_devices device node GUID ------ ---------------- mlx5_0 0c42a1030065fd82 mlx5_1 0c42a1030065fd83<o:p></o:p> </div> <div> # ibv_devinfo hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:0065:fd82 sys_image_guid: 0c42:a103:0065:fd82 vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:0065:fd83 sys_image_guid: 0c42:a103:0065:fd82 vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet<o:p></o:p> </div> <div> # mst status -v MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded PCI devices: ------------ DEVICE_TYPE MST PCI RDMA NET NUMA ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1 mlx5_1 net-enp1s0f1 -1 ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 mlx5_0 net-bond0 -1 <o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> DPDK's testpmd application sees and likes the 01:00.1: /root/Dev/dpdk/build/app/dpdk-testpmd --proc-type primary --in-memory --log-level 7 -n 4 --allow 01:00.1,class=eth -- -i EAL: Detected CPU lcores: 16 EAL: Detected NUMA nodes: 1 EAL: Detected static linkage of DPDK EAL: Selected IOVA mode 'PA' EAL: No free 2048 kB hugepages reported on node 0 EAL: VFIO support initialized EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0) mlx5_net: No available register for sampler. TELEMETRY: No legacy callbacks, legacy socket not created Interactive-mode selected testpmd: create a new mbuf pool <mb_pool_0>: n=267456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. Configuring Port 0 (socket 0) Port 0: 0C:42:A1:65:FD:83 Checking link statuses... Done testpmd> show port info 0 ********************* Infos for port 0 ********************* MAC address: 0C:42:A1:65:FD:83 Device name: 01:00.1 Driver name: mlx5_pci Firmware-version: 14.32.1010 Devargs: class=eth Connect to socket: 0 memory allocation on the socket: 0 Link status: up Link speed: 10 Gbps Link duplex: full-duplex Autoneg status: On MTU: 1500 Promiscuous mode: enabled Allmulticast mode: disabled Maximum number of MAC addresses: 128 Maximum number of MAC addresses of hash filtering: 0 VLAN offload: strip off, filter off, extend off, qinq strip off Hash key size in bytes: 40 Redirection table size: 1 Supported RSS offload flow types: ipv4 ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv6 ipv6-frag ipv6-tcp ipv6-udp ipv6-other ipv6-ex ipv6-tcp-ex ipv6-udp-ex user defined 60 user defined 61 user defined 62 user defined 63 Minimum size of RX buffer: 32 Maximum configurable length of RX packet: 65536 Maximum configurable size of LRO aggregated packet: 65280 Current number of RX queues: 1 Max possible RX queues: 1024 Max possible number of RXDs per queue: 65535 Min possible number of RXDs per queue: 0 RXDs number alignment: 1 Current number of TX queues: 1 Max possible TX queues: 1024 Max possible number of TXDs per queue: 65535 Min possible number of TXDs per queue: 0 TXDs number alignment: 1 Max segment number per packet: 40 Max segment number per MTU/TSO: 40 Device capabilities: 0x14( RXQ_SHARE FLOW_SHARED_OBJECT_KEEP ) Switch name: 01:00.1 Switch domain Id: 0 Switch Port Id: 65535 Switch Rx domain: 0 testpmd> <o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> # lsmod | egrep "(mlx|ib)" | sort ib_cm 53248 2 rdma_cm,ib_ipoib ib_core 368640 8 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm ib_ipoib 135168 0 ib_umad 24576 0 ib_uverbs 139264 2 rdma_ucm,mlx5_ib libahci 36864 1 ahci libcrc32c 16384 2 btrfs,raid456 mlx5_core 1634304 1 mlx5_ib mlx5_ib 397312 0 mlx_compat 69632 11 rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core mlxdevm 172032 1 mlx5_core mlxfw 32768 1 mlx5_core pci_hyperv_intf 16384 1 mlx5_core psample 20480 1 mlx5_core tls 94208 2 bonding,mlx5_core<o:p></o:p> </div> </div> </div> </div> </body> </html>