<div dir="ltr">Hello,<br><br><div>Thank you for your answer.</div><div>The short patch I joined with my first mail was just a rough example to report what I tested. I believe you know the driver's code better than I do, so I wouldn't be opposed to see you fix this issue.</div><div>Thank you in advance.</div><div><br></div><div>Regards,</div><div>Edwin Brossette.<br></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Wed, Mar 5, 2025 at 10:17 AM Slava Ovsiienko <<a href="mailto:viacheslavo@nvidia.com">viacheslavo@nvidia.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-3925878757954808893"> <div style="overflow-wrap: break-word;" lang="EN-US"> <div class="m_-3925878757954808893WordSection1"> <p class="MsoNormal"><span style="font-size:11pt">Hi, Edwin<u></u><u></u></span></p> <p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p> <p class="MsoNormal"><span style="font-size:11pt">Thank you for the patch.<u></u><u></u></span></p> <p class="MsoNormal"><span style="font-size:11pt">You are quite right, “sh-></span>cdev->config.hca_attr.log_max_wq_sz” is not set if DevX is disengaged.<u></u><u></u></p> <p class="MsoNormal">I found some other places where the uninitialized “log_max_wq_sz” might be used.<br> So. I’d rather prefer to configure the “log_max_wq_sz” for IBV case as well, instead of just fixing mlx5_dev_infos_get().<u></u><u></u></p> <p class="MsoNormal"><br> There is the property in “priv->sh->dev_cap.max_qp_wr”, it reflects the max number of descriptors if rdma_core is used.<u></u><u></u></p> <p class="MsoNormal">Would you like to update your patch with this? Or would you prefer me to do it ?<br> <br> <u></u><u></u></p> <p class="MsoNormal">With best regards,<br> Slava<u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p> <p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p> <div style="border-color:currentcolor currentcolor currentcolor blue;border-style:none none none solid;border-width:medium medium medium 1.5pt;padding:0cm 0cm 0cm 4pt"> <div> <div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm"> <p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> Edwin Brossette <<a href="mailto:edwin.brossette@6wind.com" target="_blank">edwin.brossette@6wind.com</a>> <br> <b>Sent:</b> Wednesday, February 12, 2025 4:34 PM<br> <b>To:</b> Slava Ovsiienko <<a href="mailto:viacheslavo@nvidia.com" target="_blank">viacheslavo@nvidia.com</a>><br> <b>Cc:</b> Asaf Penso <<a href="mailto:asafp@nvidia.com" target="_blank">asafp@nvidia.com</a>>; <a href="mailto:igootorov@gmail.com" target="_blank">igootorov@gmail.com</a>; Laurent Hardy <<a href="mailto:laurent.hardy@6wind.com" target="_blank">laurent.hardy@6wind.com</a>>; Olivier Matz <<a href="mailto:olivier.matz@6wind.com" target="_blank">olivier.matz@6wind.com</a>>; Didier Pallard <<a href="mailto:didier.pallard@6wind.com" target="_blank">didier.pallard@6wind.com</a>>; Jean-Mickael Guerin <<a href="mailto:jmg@6wind.com" target="_blank">jmg@6wind.com</a>>; <a href="mailto:dev@dpdk.org" target="_blank">dev@dpdk.org</a><br> <b>Subject:</b> Re: net/mlx5: wrong Rx/Tx descriptor limits when DevX is off<u></u><u></u></span></p> </div> </div> <p class="MsoNormal"><u></u> <u></u></p> <div> <div> <p class="MsoNormal">Hello,<u></u><u></u></p> </div> <div> <p class="MsoNormal"><u></u> <u></u></p> </div> <div> <p class="MsoNormal">Sorry for bothering you again.<u></u><u></u></p> </div> <div> <p class="MsoNormal">May I inquire if this issue is still being worked on ? <u></u> <u></u></p> </div> <div> <p class="MsoNormal">If so, when can I expect to see a fix ?<u></u><u></u></p> </div> <div> <p class="MsoNormal"><u></u> <u></u></p> </div> <div> <p class="MsoNormal">Best regards,<u></u><u></u></p> </div> <div> <p class="MsoNormal">Edwin Brossette<u></u><u></u></p> </div> </div> <p class="MsoNormal"><u></u> <u></u></p> <div> <div> <p class="MsoNormal">On Mon, Dec 23, 2024 at 2:09<span style="font-family:"Arial",sans-serif"> </span>PM Slava Ovsiienko <<a href="mailto:viacheslavo@nvidia.com" target="_blank">viacheslavo@nvidia.com</a>> wrote:<u></u><u></u></p> </div> <blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> <div> <p class="MsoNormal"><span style="font-size:11pt">Confirm, it’s a bug, IIUC was introduced by reporting function update.<br> AFAIK, we do not test with non-DevX environment anymore, so missed this.</span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt">Fix should be provided.</span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt"> </span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt">With best regards,</span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt">Slava</span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt"> </span><u></u><u></u></p> <div style="border-style:none none none solid;border-width:medium medium medium 1.5pt;padding:0cm 0cm 0cm 4pt;border-color:currentcolor currentcolor currentcolor blue"> <div> <div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm;border-color:currentcolor"> <p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> Asaf Penso <<a href="mailto:asafp@nvidia.com" target="_blank">asafp@nvidia.com</a>> <br> <b>Sent:</b> Sunday, December 22, 2024 9:39 AM<br> <b>To:</b> <a href="mailto:igootorov@gmail.com" target="_blank">igootorov@gmail.com</a>; Slava Ovsiienko <<a href="mailto:viacheslavo@nvidia.com" target="_blank">viacheslavo@nvidia.com</a>><br> <b>Cc:</b> Laurent Hardy <<a href="mailto:laurent.hardy@6wind.com" target="_blank">laurent.hardy@6wind.com</a>>; Olivier Matz <<a href="mailto:olivier.matz@6wind.com" target="_blank">olivier.matz@6wind.com</a>>; Didier Pallard <<a href="mailto:didier.pallard@6wind.com" target="_blank">didier.pallard@6wind.com</a>>; Jean-Mickael Guerin <<a href="mailto:jmg@6wind.com" target="_blank">jmg@6wind.com</a>>; Edwin Brossette <<a href="mailto:edwin.brossette@6wind.com" target="_blank">edwin.brossette@6wind.com</a>>; <a href="mailto:dev@dpdk.org" target="_blank">dev@dpdk.org</a><br> <b>Subject:</b> RE: net/mlx5: wrong Rx/Tx descriptor limits when DevX is off</span><u></u><u></u></p> </div> </div> <p class="MsoNormal"> <u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt">Hello Igor and Slava,</span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt">Can you please check out this issue?</span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt"> </span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt">Regards,</span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt">Asaf Penso</span><u></u><u></u></p> <p class="MsoNormal"><span style="font-size:11pt"> </span><u></u><u></u></p> <div style="border-style:none none none solid;border-width:medium medium medium 1.5pt;padding:0cm 0cm 0cm 4pt;border-color:currentcolor currentcolor currentcolor blue"> <div> <div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm;border-color:currentcolor"> <p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> Edwin Brossette <<a href="mailto:edwin.brossette@6wind.com" target="_blank">edwin.brossette@6wind.com</a>> <br> <b>Sent:</b> Friday, 20 December 2024 19:06<br> <b>To:</b> <a href="mailto:dev@dpdk.org" target="_blank">dev@dpdk.org</a><br> <b>Cc:</b> Laurent Hardy <<a href="mailto:laurent.hardy@6wind.com" target="_blank">laurent.hardy@6wind.com</a>>; Olivier Matz <<a href="mailto:olivier.matz@6wind.com" target="_blank">olivier.matz@6wind.com</a>>; Didier Pallard <<a href="mailto:didier.pallard@6wind.com" target="_blank">didier.pallard@6wind.com</a>>; Jean-Mickael Guerin <<a href="mailto:jmg@6wind.com" target="_blank">jmg@6wind.com</a>><br> <b>Subject:</b> net/mlx5: wrong Rx/Tx descriptor limits when DevX is off</span><u></u><u></u></p> </div> </div> <p class="MsoNormal"> <u></u><u></u></p> <div> <p class="MsoNormal" style="margin-bottom:12pt">Hello,<br> <br> I have run into a regression following an update to stable dpdk-24.11 with a number of my Mellanox cx4/5/6 nics. This regression occurs with all nics in my lab which have DevX disabled: using mstconfig utility, I can see the flag UCTX_EN is not set.<br> <br> Mainly, the issue is that the ports cannot be started, with the following error logs in the journal:<u></u><u></u></p> <div style="margin-left:30pt"> <p class="MsoNormal">Set nb_rxd=1 (asked=512) for port=0<br> Set nb_txd=1 (asked=512) for port=0<br> starting port 0<br> Initializing port 0 [7c:fe:90:65:e6:54]<br> port 0: ntfp1 (mlx5_pci)<br> nb_rxq=2 nb_txq=2<br> rxq0=c9 rxq1=c25<br> txq0=c9 txq1=c25<br> port 0: rx_scatter=0 tx_scatter=0 max_rx_frame=1526<br> mlx5_net: port 0 number of descriptors requested for Tx queue 0 must be higher than MLX5_TX_COMP_THRESH, using 33 instead of 1<br> mlx5_net: port 0 increased number of descriptors in Tx queue 0 to the next power of two (64)<br> mlx5_net: port 0 number of descriptors requested for Tx queue 1 must be higher than MLX5_TX_COMP_THRESH, using 33 instead of 1<br> mlx5_net: port 0 increased number of descriptors in Tx queue 1 to the next power of two (64)<br> mlx5_net: Port 0 Rx queue 0 CQ creation failure.<br> mlx5_net: port 0 Rx queue allocation failed: Cannot allocate memory<br> rte_eth_dev_start(port 0) failed, error=-12<br> Failed to start port 0, set link down<br> Failed to start port 0<u></u><u></u></p> </div> <p class="MsoNormal" style="margin-bottom:12pt"><br> Looking more precisely into the problem, it appears that the number of Rx and Tx descriptors configured for my queues is 1. This happens because mlx5_dev_infos_get() return a limit of 1 for both Rx and Tx, which is unexpected. I identified this patch to be responsible for the regression:<br> <br> 4c3d7961d9002: net/mlx5: fix reported Rx/Tx descriptor limits<br> <a href="https://git.dpdk.org/dpdk/commit/?id=4c3d7961d9002bb715a8ee76bcf464d633316d4c" target="_blank">https://git.dpdk.org/dpdk/commit/?id=4c3d7961d9002bb715a8ee76bcf464d633316d4c</a><br> <br> After doing some debugging, I noticed that hca_attr.log_max_wq_sz is never configured. This should be done in mlx5_devx_cmd_query_hca_attr() which is called in this bit of code:<br> <br> <a href="https://git.dpdk.org/dpdk/tree/drivers/common/mlx5/mlx5_common.c#n681" target="_blank">https://git.dpdk.org/dpdk/tree/drivers/common/mlx5/mlx5_common.c#n681</a><u></u><u></u></p> <div style="margin-left:30pt"> <p class="MsoNormal">/*<br> * When CTX is created by Verbs, query HCA attribute is unsupported.<br> * When CTX is imported, we cannot know if it is created by DevX or<br> * Verbs. So, we use query HCA attribute function to check it.<br> */<br> if (cdev->config.devx || cdev->config.device_fd != MLX5_ARG_UNSET) {<u></u><u></u></p> </div> <div style="margin-left:30pt"> <div style="margin-left:30pt"> <p class="MsoNormal">/* Query HCA attributes. */<br> ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &cdev->config.hca_attr);<br> if (ret) {<u></u><u></u></p> </div> </div> <div style="margin-left:30pt"> <div style="margin-left:30pt"> <div style="margin-left:30pt"> <p class="MsoNormal">DRV_LOG(ERR, "Unable to read HCA caps in DevX mode.");<br> rte_errno = ENOTSUP;<br> goto error;<u></u><u></u></p> </div> </div> </div> <div style="margin-left:30pt"> <div style="margin-left:30pt"> <p class="MsoNormal">}<br> cdev->config.devx = 1;<u></u><u></u></p> </div> </div> <div style="margin-left:30pt"> <p class="MsoNormal">}<br> DRV_LOG(DEBUG, "DevX is %ssupported.", cdev->config.devx ? "" : "NOT ");<u></u><u></u></p> </div> <p class="MsoNormal"> <u></u><u></u></p> <div> <p class="MsoNormal">I deduced that following the above patch, the correct value for maximum Rx and Tx descriptors will only be set if DevX is enabled (see the if condition on cdev->config.devx). If it is disabled, then maximum Rx and Tx descriptors will be 1, which will make the ports fail to start. Perhaps we should keep the previous default value (65535) if config.devx == 0 (DevX off)? This could be done like this, for example:<br> <br> diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c<br> index 7708a0b80883..8ba3eb4a32de 100644<br> --- a/drivers/net/mlx5/mlx5_ethdev.c<br> +++ b/drivers/net/mlx5/mlx5_ethdev.c<br> @@ -359,10 +359,12 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)<br> info->flow_type_rss_offloads = ~MLX5_RSS_HF_MASK;<br> mlx5_set_default_params(dev, info);<br> mlx5_set_txlimit_params(dev, info);<br> - info->rx_desc_lim.nb_max =<br> - 1 << priv->sh->cdev->config.hca_attr.log_max_wq_sz;<br> - info->tx_desc_lim.nb_max =<br> - 1 << priv->sh->cdev->config.hca_attr.log_max_wq_sz;<br> + if (priv->sh->cdev->config.devx) {<br> + info->rx_desc_lim.nb_max =<br> + 1 << priv->sh->cdev->config.hca_attr.log_max_wq_sz;<br> + info->tx_desc_lim.nb_max =<br> + 1 << priv->sh->cdev->config.hca_attr.log_max_wq_sz;<br> + }<br> if (priv->sh->cdev->config.hca_attr.mem_rq_rmp &&<br> priv->obj_ops.rxq_obj_new == devx_obj_ops.rxq_obj_new)<br> info->dev_capa |= RTE_ETH_DEV_CAPA_RXQ_SHARE;<br> <br> Thanks in advance for your help.<br> <br> Regards,<br> Edwin Brossette.<u></u><u></u></p> <div> <p class="MsoNormal" style="margin-bottom:12pt"> <u></u><u></u></p> </div> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> </div> </div> </div> </div></blockquote></div>