<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="en-SE" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hi Slava and Erez and thanks for your answers,<o:p></o:p> <o:p> </o:p> Regarding the firmware, I’ve also deployed in a different OpenShift cluster were I see the exact same issue but with a different Mellanox NIC:<o:p></o:p> <o:p> </o:p> Mellanox Technologies MT2892 Family - ConnectX-6 DX 2-port 100GbE QSFP56 PCIe Adapter<o:p></o:p> driver: mlx5_core<o:p></o:p> version: 5.0-0 firmware-version: 22.36.1010 (DEL0000000027)<o:p></o:p> <o:p> </o:p> From what I can see the firmware is relatively new on that one?<o:p></o:p> <o:p> </o:p> I tried setting dv_flow_en=0 (and saw that it was propagated to config->dv_flow_en) but it didn’t seem to help.<o:p></o:p> <o:p> </o:p> Erez, I’m not sure what you mean by shared or non-shared mode in this case, however it seems it could be related to the fact that the container is running in a separate network namespace. Because the hw_counter directory is available on the host (cluster node), but not in the pod container.<o:p></o:p> <o:p> </o:p> Best regards,<o:p></o:p> Daniel<o:p></o:p> <o:p> </o:p> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Erez Ferber <erezferber@gmail.com> Sent: Monday, 5 June 2023 12:29 To: Slava Ovsiienko <viacheslavo@nvidia.com> Cc: Daniel Östman <daniel.ostman@ericsson.com>; users@dpdk.org; Matan Azrad <matan@nvidia.com>; maxime.coquelin@redhat.com; david.marchand@redhat.com Subject: Re: mlx5: imissed / out_of_buffer counter always 0<o:p></o:p> </div> <o:p> </o:p> <div> <div id="gmail-:uv"> Hi Daniel,<o:p></o:p> <div> <o:p> </o:p> </div> <div> is the container running in shared or non-shared mode ? <o:p></o:p> </div> <div> For shared mode, I assume the kernel sysfs counters which DPDK relies on for imissed/out_of_buffer are not exposed.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> Best regards,<o:p></o:p> </div> <div> Erez<o:p></o:p> </div> </div> </div> <o:p> </o:p> <div> <div> On Fri, 2 Jun 2023 at 18:07, Slava Ovsiienko <<a href="mailto:viacheslavo@nvidia.com">viacheslavo@nvidia.com</a>> wrote:<o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> <div> Hi, Daniel<o:p></o:p> <o:p></o:p> I would recommend to take the following action:<o:p></o:p> - update the firmware, 16.33.xxxx looks to be outdated a little bit. Please, try 16.35.1012 or later. mlx5_glue->devx_obj_create might succeed with the newer FW.<o:p></o:p> - try to specify dv_flow_en=0 devarg, it forces mlx5 PMD to use rdma_core library for queue management and kernel driver will be aware about Rx queues being created and attach them to the kernel counter set<o:p></o:p> <o:p></o:p> With best regards, Slava<o:p></o:p> <o:p></o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Daniel Östman <<a href="mailto:daniel.ostman@ericsson.com" target="_blank">daniel.ostman@ericsson.com</a>> Sent: Friday, June 2, 2023 3:59 PM To: <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> Cc: Matan Azrad <<a href="mailto:matan@nvidia.com" target="_blank">matan@nvidia.com</a>>; Slava Ovsiienko <<a href="mailto:viacheslavo@nvidia.com" target="_blank">viacheslavo@nvidia.com</a>>; <a href="mailto:maxime.coquelin@redhat.com" target="_blank">maxime.coquelin@redhat.com</a>; <a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a> Subject: mlx5: imissed / out_of_buffer counter always 0<o:p></o:p> </div> </div> <o:p></o:p> Hi,<o:p></o:p> <o:p></o:p> I’m deploying a containerized DPDK application in an OpenShift Kubernetes environment using DPDK 21.11.3.<o:p></o:p> The application uses a Mellanox ConnectX-5 100G NIC through VFs. <o:p></o:p> <o:p></o:p> The problem I have is that the ETH stats counter imissed (which seems to be mapped to “out_of_buffer” internally in mlx5 PMD driver) is 0 when I don’t expect it to be, i.e. when the application doesn’t read the packets fast enough.<o:p></o:p> <o:p></o:p> Using GDB I can see that it tries to access the counter through /sys/class/infiniband/mlx5_99/ports/1/hw_counters/out_of_buffer but the hw_counters directory is missing so it will just return a zero value. I don’t know why it is missing.<o:p></o:p> When looking at mlx5_os_read_dev_stat() I can see that there is an alternative way of reading the counter, through mlx5_devx_cmd_queue_counter_query() but under the condition that priv->q_counters are set.<o:p></o:p> It doesn’t get set in my case because mlx5_glue->devx_obj_create() fails (errno 22) in mlx5_devx_cmd_queue_counter_alloc().<o:p></o:p> <o:p></o:p> Have I missed something?<o:p></o:p> <o:p></o:p> NIC info:<o:p></o:p> Mellanox Technologies MT27800 Family [ConnectX-5] - 100Gb 2-port QSFP28 MCX516A-CCHT driver: mlx5_core version: 5.0-0 firmware-version: 16.33.1048 (MT_0000000417)<o:p></o:p> <o:p></o:p> Please let me know if I need to provide more information.<o:p></o:p> <o:p></o:p> Best regards,<o:p></o:p> Daniel<o:p></o:p> <o:p></o:p> </div> </div> </div> </div> </blockquote> </div> </div> </body> </html>