<div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div style="margin: 0;">Hi Slava, matan:</div><div style="margin: 0;">Thanks for the info! </div><div style="margin: 0;"><br></div><div style="margin: 0;">I read the failsafe code briefly. here are my superficial undestanding. if there is anything worng, please help correct</div><div style="margin: 0;">failsafe seems to be a virtual device, which can be created by using --vdev args when dpdk app is started:</div><div style="margin: 0;">./app --vdev 'net_failsafe0,mac=xx:xx, dev(84:00:00)'</div><div style="margin: 0;">the above command will create a net_failsafe0 device, its sub device is 84:00:00, this can be a mlx5 physical NIC. </div><div style="margin: 0;">By adding the mlx5 physical NIC as a failsafe subdevice, the mlx5 NIC can hotplug without requiring me to modify my app? </div><div style="margin: 0;">I also found the dpdk community documentation:</div><div style="margin: 0;"><a href="https://doc.dpdk.org/guides-20.02/nics/fail_safe.html">53. Fail-safe poll mode driver library бк Data Plane Development Kit 20.02.1 documentation (dpdk.org)</a></div><p style="margin: 0;"><br></p><p>At 2023-08-17 13:13:18, "Slava Ovsiienko" <viacheslavo@nvidia.com> wrote:</p><blockquote id="isReplyContent" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><div xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml"> <style></style> <div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Hi, Jinag<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">5 cents to add: please see how FailSafe PMD handles device removal events and probes the devices periodically.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">This is good example to understand how device hotplug works (Failsafe uses rte_eal_hotplug_add())<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Also, to add the device the previous instance should closed (RTE_ETH_EVENT_INTR_RMV) shoud be handled accordingly.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">With best regards,<br> Slava<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"> Matan Azrad <matan@nvidia.com> <br> <b>Sent:</b> Monday, August 14, 2023 9:41 AM<br> <b>To:</b> jinag <15720603159@163.com>; users@dpdk.org; Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com><br> <b>Subject:</b> RE: Does the mlx5 NIC support reloading<o:p></o:p></span></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Hi Jinag<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">After plugging out the device from the bus, you need to get the event </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white">RTE_ETH_EVENT_INTR_RMV</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white">.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white">You need to listen to this event and close the port when you see it.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white">After plugging in the device, you need to scan the bus again and attach the mlx5 device, so it will be probed again and a new ethdev port will be created.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white">Then, you need to reconfigure the port, as regular, and reuse the device.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white">Thanks,<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:white">Matan<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> jinag <<a href="mailto:15720603159@163.com">15720603159@163.com</a>> <br> <b>Sent:</b> Monday, 14 August 2023 6:07<br> <b>To:</b> <a href="mailto:users@dpdk.org">users@dpdk.org</a>; Matan Azrad <<a href="mailto:matan@nvidia.com">matan@nvidia.com</a>>; Shahaf Shuler <<a href="mailto:shahafs@nvidia.com">shahafs@nvidia.com</a>>; Slava Ovsiienko <<a href="mailto:viacheslavo@nvidia.com">viacheslavo@nvidia.com</a>><br> <b>Subject:</b> Does the mlx5 NIC support reloading<o:p></o:p></span></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <table class="MsoNormalTable ntes_not_fresh_table" border="1" cellpadding="0" style="background:#FFEB9C"> <tbody> <tr> <td style="padding:.75pt .75pt .75pt .75pt"> <p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black"> </span><o:p></o:p></p> </td> </tr> </tbody> </table> <p class="MsoNormal"><o:p> </o:p></p> <div> <div> <div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Hi <o:p></o:p></span></p> </div> <div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">I am verifying the reload function of the mlx5 nic based on dpdk 19.11:<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">echo 1 > /sys/bus/pci/devices/$pci_address/remove<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">echo 1 > /sys/bus/pci/rescan<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">rte_bus_probe(); <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">rte_eth_dev_stop(); <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">rte_eth_dev_start(); <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> net_mlx5: port 0 TX queue 0 CQ creation failure<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"> net_mlx5: port 0 TX queue allocation failed: cannot allocate memory<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">The nic cannot be reinitialized.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">I am not sure if the above operation is correct. Could you please tell me if the mlx5 nic supports reloading(for example, the network card is down during normal operation) and which dpdk functions need to be called.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Thanks!<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </blockquote></div>