<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="ZH-CN" link="#0563C1" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hi Alex.,<o:p></o:p> <o:p> </o:p> Is this 0x22/0x2d telling the error type?<o:p></o:p> Unexpected CQE error syndrome 0x22<o:p></o:p> Unexpected CQE error syndrome 0x2d<o:p></o:p> I find below, but it seems it is for windows (drivers\common\mlx5\windows\mlx5_win_defs.h), but I’m using linux.<o:p></o:p> enum {<o:p></o:p> MLX5_CQE_SYNDROME_LOCAL_LENGTH_ERR = 0x01,<o:p></o:p> MLX5_CQE_SYNDROME_LOCAL_QP_OP_ERR = 0x02,<o:p></o:p> MLX5_CQE_SYNDROME_LOCAL_PROT_ERR = 0x04,<o:p></o:p> MLX5_CQE_SYNDROME_WR_FLUSH_ERR = 0x05,<o:p></o:p> MLX5_CQE_SYNDROME_MW_BIND_ERR = 0x06,<o:p></o:p> MLX5_CQE_SYNDROME_BAD_RESP_ERR = 0x10,<o:p></o:p> MLX5_CQE_SYNDROME_LOCAL_ACCESS_ERR = 0x11,<o:p></o:p> MLX5_CQE_SYNDROME_REMOTE_INVAL_REQ_ERR = 0x12,<o:p></o:p> MLX5_CQE_SYNDROME_REMOTE_ACCESS_ERR = 0x13,<o:p></o:p> MLX5_CQE_SYNDROME_REMOTE_OP_ERR = 0x14,<o:p></o:p> MLX5_CQE_SYNDROME_TRANSPORT_RETRY_EXC_ERR = 0x15,<o:p></o:p> MLX5_CQE_SYNDROME_RNR_RETRY_EXC_ERR = 0x16,<o:p></o:p> MLX5_CQE_SYNDROME_REMOTE_ABORTED_ERR = 0x22,<o:p></o:p> };<o:p></o:p> <o:p> </o:p> Do I understand correctly with below commit “net/mlx5: ignore non-critical syndromes for Rx queue”<o:p></o:p> syndrome 0x22/0x2d will not cause rx queue reset?<o:p></o:p> <o:p> </o:p> commit aa67ed3084588e6ca12e9709a6cab021f0ffeba7<o:p></o:p> Author: Alexander Kozyrev <a href="mailto:akozyrev@nvidia.com">akozyrev@nvidia.com</a><o:p></o:p> Date: Fri Jan 27 05:22:43 2023 +0200<o:p></o:p> <o:p> </o:p> net/mlx5: ignore non-critical syndromes for Rx queue<o:p></o:p> <o:p> </o:p> For non-fatal syndromes like LOCAL_LENGTH_ERR, the Rx queue reset<o:p></o:p> shouldn't be triggered. Rx queue could continue with the next packets<o:p></o:p> without any recovery. Only three syndromes warrant Rx queue reset:<o:p></o:p> LOCAL_QP_OP_ERR, LOCAL_PROT_ERR and WR_FLUSH_ERR.<o:p></o:p> Do not initiate a Rx queue reset in any other cases.<o:p></o:p> Skip all non-critical error CQEs and continue with packet processing.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <div> Br, Xiaoping<o:p></o:p> </div> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Alexander Kozyrev <akozyrev@nvidia.com> Sent: 2023年7月27日 20:43 To: Xiaoping Yan (NSB) <xiaoping.yan@nokia-sbell.com> Cc: Matan Azrad <matan@nvidia.com>; users@dpdk.org Subject: [External] RE: dpdk mlx5 driver crash in rxq_cq_decompress_v<o:p></o:p> </div> </div> <o:p> </o:p> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%"> <tbody> <tr> <td style="background:#FFB900;padding:5.0pt 2.0pt 5.0pt 2.0pt"> <o:p></o:p> </td> <td width="100%" style="width:100.0%;background:#FFF8E5;padding:5.0pt 4.0pt 5.0pt 12.0pt"> <div> CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See <a href="http://nok.it/nsb">http://nok.it/nsb</a> for additional information.<o:p></o:p> </div> </td> </tr> </tbody> </table> <o:p></o:p> Hi Xiaoping, looks like sometimes you get a packet that exceeds VF MTU resulting in an error CQE and subsequent crash.<o:p></o:p> Could you please cherry-pick “547b239a21 net/mlx5: ignore non-critical syndromes for Rx queue” to your DPDK version?<o:p></o:p> This should fix the crash by not resetting the Rx queue in this scenario. Alternatively, you can set VF MTU to 9000 to match PF.<o:p></o:p> <o:p> </o:p> Regards,<o:p></o:p> Alex<o:p></o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Xiaoping Yan (NSB) <<a href="mailto:xiaoping.yan@nokia-sbell.com">xiaoping.yan@nokia-sbell.com</a>> Sent: Tuesday, July 11, 2023 10:24 PM To: Alexander Kozyrev <<a href="mailto:akozyrev@nvidia.com">akozyrev@nvidia.com</a>> Cc: Matan Azrad <<a href="mailto:matan@nvidia.com">matan@nvidia.com</a>>; <a href="mailto:users@dpdk.org"> users@dpdk.org</a> Subject: RE: dpdk mlx5 driver crash in rxq_cq_decompress_v<o:p></o:p> </div> </div> <o:p> </o:p> Hi Alex,<o:p></o:p> <o:p> </o:p> PF MTU is 9000, VF MTU is 2000 (tried with 1500 also and get same crash).<o:p></o:p> Here is the test topology.<o:p></o:p> Traffic pattern:<o:p></o:p> downlink: gtpu packet length 1236, throughput: 3.65Gbps<o:p></o:p> uplink: gtpu packet length 302, throughput 0.47Gbps<o:p></o:p> <o:p> </o:p> Crash seen in the l2hicu Test2 container.<o:p></o:p> This commit (547b239a21) you mentioned in the other mail chain is not included in my dpdk version.<o:p></o:p> <o:p> </o:p> <img border="0" width="735" height="369" style="width:7.6527in;height:3.8402in" id="Picture_x0020_1" src="cid:image001.png@01D9FDC7.0523ECE0"><o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div> Br, Xiaoping<o:p></o:p> </div> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Alexander Kozyrev <<a href="mailto:akozyrev@nvidia.com">akozyrev@nvidia.com</a>> Sent: 2023年7月12日 4:48 To: Xiaoping Yan (NSB) <<a href="mailto:xiaoping.yan@nokia-sbell.com">xiaoping.yan@nokia-sbell.com</a>> Cc: Matan Azrad <<a href="mailto:matan@nvidia.com">matan@nvidia.com</a>>; <a href="mailto:users@dpdk.org"> users@dpdk.org</a> Subject: [External] RE: dpdk mlx5 driver crash in rxq_cq_decompress_v<o:p></o:p> </div> </div> <o:p> </o:p> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%"> <tbody> <tr> <td style="background:#FFB900;padding:5.0pt 2.0pt 5.0pt 2.0pt"> <o:p></o:p> </td> <td width="100%" style="width:100.0%;background:#FFF8E5;padding:5.0pt 4.0pt 5.0pt 12.0pt"> <div> CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See <a href="http://nok.it/nsb">http://nok.it/nsb</a> for additional information.<o:p></o:p> </div> </td> </tr> </tbody> </table> <o:p></o:p> Hi Xiaoping, I cannot reproduce the issue locally, all the fixes for CQE recovery are the part of 22.11.2 already.<o:p></o:p> Would you mind sharing more information about your setup, test-case and traffic characteristics? Do you have VF/PF MTU mismatch?<o:p></o:p> <o:p> </o:p> Regards,<o:p></o:p> Alex<o:p></o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Xiaoping Yan (NSB) <<a href="mailto:xiaoping.yan@nokia-sbell.com">xiaoping.yan@nokia-sbell.com</a>> Sent: Tuesday, July 4, 2023 10:57 PM To: Alexander Kozyrev <<a href="mailto:akozyrev@nvidia.com">akozyrev@nvidia.com</a>> Subject: RE: dpdk mlx5 driver crash in rxq_cq_decompress_v<o:p></o:p> </div> </div> <o:p> </o:p> Hi Alex,<o:p></o:p> <o:p> </o:p> Here is the CQE<o:p></o:p> <o:p> </o:p> <div> Br, Xiaoping<o:p></o:p> </div> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Alexander Kozyrev <<a href="mailto:akozyrev@nvidia.com">akozyrev@nvidia.com</a>> Sent: 2023年7月5日 9:45 To: Matan Azrad <<a href="mailto:matan@nvidia.com">matan@nvidia.com</a>>; Xiaoping Yan (NSB) <<a href="mailto:xiaoping.yan@nokia-sbell.com">xiaoping.yan@nokia-sbell.com</a>>; <a href="mailto:users@dpdk.org">users@dpdk.org</a>; Dekel Peled <<a href="mailto:dekelp@nvidia.com">dekelp@nvidia.com</a>> Subject: [External] RE: dpdk mlx5 driver crash in rxq_cq_decompress_v<o:p></o:p> </div> </div> <o:p> </o:p> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%"> <tbody> <tr> <td style="background:#FFB900;padding:5.0pt 2.0pt 5.0pt 2.0pt"> <o:p></o:p> </td> <td width="100%" style="width:100.0%;background:#FFF8E5;padding:5.0pt 4.0pt 5.0pt 12.0pt"> <div> CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See <a href="http://nok.it/nsb">http://nok.it/nsb</a> for additional information.<o:p></o:p> </div> </td> </tr> </tbody> </table> <o:p></o:p> Hi Xiaoping, could you please forward the error CQE dump to me?<o:p></o:p> Would you mind elaborating more on your traffic pattern and test case scenario?<o:p></o:p> The following commit supposed to ignore MTU mismatch error between VF and PF:<o:p></o:p> 547b239a21 net/mlx5: ignore non-critical syndromes for Rx queue<o:p></o:p> <o:p> </o:p> Regards,<o:p></o:p> Alex<o:p></o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Matan Azrad <<a href="mailto:matan@nvidia.com">matan@nvidia.com</a>> Sent: Sunday, July 2, 2023 11:35 PM To: Xiaoping Yan (NSB) <<a href="mailto:xiaoping.yan@nokia-sbell.com">xiaoping.yan@nokia-sbell.com</a>>; <a href="mailto:users@dpdk.org">users@dpdk.org</a>; Dekel Peled <<a href="mailto:dekelp@nvidia.com">dekelp@nvidia.com</a>>; Alexander Kozyrev <<a href="mailto:akozyrev@nvidia.com">akozyrev@nvidia.com</a>> Subject: Re: dpdk mlx5 driver crash in rxq_cq_decompress_v<o:p></o:p> </div> </div> <o:p> </o:p> + <a id="OWAAM000001" href="mailto:akozyrev@nvidia.com">@Alexander Kozyrev</a> to suggest. <o:p></o:p> <div> <o:p> </o:p> </div> <div id="ms-outlook-mobile-signature"> קבל <a href="https://aka.ms/AAb9ysg">‏Outlook עבור Android‏</a><o:p></o:p> </div> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="1" width="98%" align="center"> </div> <div id="divRplyFwdMsg"> From: Xiaoping Yan (NSB) <<a href="mailto:xiaoping.yan@nokia-sbell.com">xiaoping.yan@nokia-sbell.com</a>> Sent: Monday, July 3, 2023 4:18:22 AM To: <a href="mailto:users@dpdk.org">users@dpdk.org</a> <<a href="mailto:users@dpdk.org">users@dpdk.org</a>>; Matan Azrad <<a href="mailto:matan@nvidia.com">matan@nvidia.com</a>>; <a href="mailto:dekelp@nvidia.com">dekelp@nvidia.com</a> <<a href="mailto:dekelp@nvidia.com">dekelp@nvidia.com</a>> Subject: RE: dpdk mlx5 driver crash in rxq_cq_decompress_v <o:p></o:p> <div> <o:p></o:p> </div> </div> <div> <table class="MsoNormalTable" border="1" cellspacing="3" cellpadding="0" style="background:#FFEB9C"> <tbody> <tr> <td style="padding:.75pt .75pt .75pt .75pt"> External email: Use caution opening links or attachments <o:p></o:p> </td> </tr> </tbody> </table> <o:p> </o:p> <div> <div> Hi,<o:p></o:p> <o:p></o:p> <a id="OWAAM4F32193598C54B96899A7C78B433AACA" href="mailto:dekelp@nvidia.com">@'dekelp@nvidia.com'</a><a id="OWAAM72C297E17CCC489B9901E36573FE6073" href="mailto:matan@nvidia.com">@'Matan Azrad'</a> Can you kindly suggest?<o:p></o:p> Thank you.<o:p></o:p> <o:p></o:p> <div> Br, Xiaoping<o:p></o:p> </div> <o:p></o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Xiaoping Yan (NSB) Sent: 2023年6月27日 12:11 To: <a href="mailto:users@dpdk.org">users@dpdk.org</a>; 'Matan Azrad' <<a href="mailto:matan@nvidia.com">matan@nvidia.com</a>>; 'dekelp@nvidia.com' <<a href="mailto:dekelp@nvidia.com">dekelp@nvidia.com</a>> Subject: dpdk mlx5 driver crash in rxq_cq_decompress_v<o:p></o:p> </div> </div> <o:p></o:p> Hi,<o:p></o:p> <o:p></o:p> dpdk version in use: 21.11.2<o:p></o:p> <o:p></o:p> Mlx5 driver crashes in rxq_cq_decompress_v in traffic test after several minutes.<o:p></o:p> Stack trace:<o:p></o:p> (gdb) bt<o:p></o:p> #0 0x00007ffff58612bc in _mm_storeu_si128 (__B=..., __P=<optimized out>)<o:p></o:p> at /usr/lib/gcc/x86_64-redhat-linux/12/include/emmintrin.h:739<o:p></o:p> #1 rxq_cq_decompress_v (rxq=rxq@entry=0x2abe5592f40, cq=cq@entry=0x2abe54fdb00, elts=elts@entry=0x2abe5594638)<o:p></o:p> at ../dpdk-21.11/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:142<o:p></o:p> #2 0x00007ffff5862c84 in rxq_burst_v (no_cq=<synthetic pointer>, err=0x7fffffffb848, pkts_n=4, pkts=<optimized out>, <o:p></o:p> rxq=0x2abe5592f40) at ../dpdk-21.11/drivers/net/mlx5/mlx5_rxtx_vec.c:349<o:p></o:p> #3 mlx5_rx_burst_vec (dpdk_rxq=0x2abe5592f40, pkts=0x7fffffffbf80, pkts_n=32) at ../dpdk-21.11/drivers/net/mlx5/mlx5_rxtx_vec.c:393<o:p></o:p> #4 0x00005555556a0f41 in rte_eth_rx_burst (nb_pkts=32, rx_pkts=0x7fffffffbf80, queue_id=0, port_id=1)<o:p></o:p> at /usr/include/rte_ethdev.h:5721<o:p></o:p> …<o:p></o:p> Attached is the error log “Unexpected CQE error syndrome…” and dump file<o:p></o:p> <o:p></o:p> I found there was a similar bug here: <a href="https://bugs.dpdk.org/show_bug.cgi?id=334"> https://bugs.dpdk.org/show_bug.cgi?id=334</a><o:p></o:p> But the fix (88c0733535d6 extend Rx completion with error handling) should already been included, as I’m using 21.11.2<o:p></o:p> Also below commit (fix to 88c0733535d6) is already included in my dpdk version.<o:p></o:p> commit 60b254e3923d007bcadbb8d410f95ad89a2f13fa<o:p></o:p> Author: Matan Azrad <a href="mailto:matan@nvidia.com">matan@nvidia.com</a><o:p></o:p> Date: Thu Aug 11 19:51:55 2022 +0300<o:p></o:p> <o:p></o:p> net/mlx5: fix Rx queue recovery mechanism<o:p></o:p> <o:p></o:p> Any suggestion?<o:p></o:p> Thank you.<o:p></o:p> <o:p></o:p> Br, Xiaoping<o:p></o:p> <o:p></o:p> </div> </div> </div> </div> </div> </div> </div> </body> </html>