<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="#0563C1" vlink="#954F72" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi, All:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I am using Mellanox ConnectX-5 and ConnectX-4 Lx with DPDK v21.11 but<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">there is a probability that the nic can't send packets.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">One condition is that the contiguous physical of hugepages allocated<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">on the host is poor. For example, if the environment is configured with<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">10GB hugepages but each hugepage is physically discontinuous, this problem<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">can be reproduced.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This problem is introduced by this patch:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">https://git.dpdk.org/dpdk/commit/?id=fec28ca0e3a93143829f3b41a28a8da933f28499.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">LOG:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">dpdk # ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0xFC0 --iova-mode pa
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">--legacy-mem -a 03:00.0 -a 03:00.1 -m 8192,0 -- -a -i --forward-mode=fwd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">--rxq=4 --txq=4 --total-num-mbufs=1000000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">EAL: Detected CPU lcores: 72<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">EAL: Detected NUMA nodes: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">EAL: Detected static linkage of DPDK<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">EAL: Multi-process socket /var/run/dpdk/rte/mp_socket<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">EAL: Selected IOVA mode 'PA'<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.0 (socket 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">mlx5_net: Default miss action is not supported.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.1 (socket 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">mlx5_net: Default miss action is not supported.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">TELEMETRY: No legacy callbacks, legacy socket not created<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Auto-start selected<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Interactive-mode selected<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Invalid fwd packet forwarding mode<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">testpmd: create a new mbuf pool <mb_pool_0>: n=1000000, size=2176, socket=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">testpmd: preferred mempool ops selected: ring_mp_mc<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Configuring Port 0 (socket 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Port 0: 28:DE:E5:AB:9D:CA<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Configuring Port 1 (socket 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Port 1: 28:DE:E5:AB:9D:CB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Checking link statuses...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Done<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Start automatic packet forwarding<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">io packet forwarding - ports=2 - cores=1 - streams=8 - NUMA support enabled, MP allocation mode: native<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Logical Core 7 (socket 0) forwards packets on 8 streams:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX P=0/Q=1 (socket 0) -> TX P=1/Q=1 (socket 0) peer=02:00:00:00:00:01<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX P=1/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX P=0/Q=2 (socket 0) -> TX P=1/Q=2 (socket 0) peer=02:00:00:00:00:01<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX P=1/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX P=0/Q=3 (socket 0) -> TX P=1/Q=3 (socket 0) peer=02:00:00:00:00:01<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX P=1/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> io packet forwarding packets/burst=32<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> nb forwarding cores=1 - nb forwarding ports=2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> port 0: RX queue number: 4 Tx queue number: 4<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Rx offloads=0x0 Tx offloads=0x10000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX queue: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX desc=4096 - RX free threshold=64<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX threshold registers: pthresh=0 hthresh=0 wthresh=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX Offloads=0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> TX queue: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> TX desc=4096 - TX free threshold=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> TX threshold registers: pthresh=0 hthresh=0 wthresh=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> TX offloads=0x10000 - TX RS bit threshold=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> port 1: RX queue number: 4 Tx queue number: 4<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Rx offloads=0x0 Tx offloads=0x10000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX queue: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX desc=4096 - RX free threshold=64<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX threshold registers: pthresh=0 hthresh=0 wthresh=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> RX Offloads=0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> TX queue: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> TX desc=4096 - TX free threshold=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> TX threshold registers: pthresh=0 hthresh=0 wthresh=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> TX offloads=0x10000 - TX RS bit threshold=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">testpmd> mlx5_net: Cannot change Tx QP state to INIT Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">mlx5_net: Cannot change Tx QP state to INIT Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">mlx5_net: Cannot change Tx QP state to INIT Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">mlx5_net: Cannot change Tx QP state to INIT Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">testpmd> mlx5_net: Cannot change Tx QP state to INIT Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">mlx5_net: Cannot change Tx QP state to INIT Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">quimlx5_net: Cannot change Tx QP state to INIT Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">mlx5_net: Cannot change Tx QP state to INIT Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">And create some files:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/var/log/dpdk_mlx5_port_0_txq_0_index_0_1883249505<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/var/log/dpdk_mlx5_port_0_txq_0_index_0_2291454530<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/var/log/dpdk_mlx5_port_0_txq_0_index_0_2880295119<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/var/log/dpdk_mlx5_port_1_txq_0_index_0_2198716197<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/var/log/dpdk_mlx5_port_1_txq_0_index_0_2498129310<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/var/log/dpdk_mlx5_port_1_txq_0_index_0_3046021743<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Unexpected CQE error syndrome 0x04 CQN = 256 SQN = 6612 wqe_counter = 0 wq_ci = 1 cq_ci = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">MLX5 Error CQ: at [0x7f6edca57000], len=16384<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">00000020: 00 00 00 01 73 65 65 6E 00 00 00 00 00 00 00 00 | ....seen........<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">00000030: 00 00 00 00 9D 00 53 04 29 00 19 D4 00 00 02 D2 | ......S.).......<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F0 | ................<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................<o:p></o:p></span></p>
</div>
</body>
</html>