<div dir="ltr"><div>Dear Harry,</div><div><br></div><div>Concern is not static linking or dynamic linking. Concern is Card is shutting down immediately after devbind. This, I am able to clearly see as the green LEDs are turning immediately off after executing devbind.</div><div>However, I have executed and the following is the observation.<br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"></div><div dir="ltr"></div><div dir="ltr"><a class="gmail_plusreply" id="plusReplyChip-0">+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</a><br></div><div dir="ltr">$ lspci | grep -i intel<br>c3:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)<br>c3:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)<br>$ sudo dpdk-devbind.py -b vfio-pci c3:00.0 c3:00.1<br>$ cat one.c <br>#include <inttypes.h><br>#include <rte_ethdev.h><br><br>int main(int argc, char *argv[])<br>{<br> unsigned nb_ports;<br><br> int ret = rte_eal_init(argc, argv);<br> if (ret < 0)<br> rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");<br><br> nb_ports = rte_eth_dev_count_avail();<br> fprintf( stderr, "%s %d nb_ports: %u\n", __func__, __LINE__, nb_ports );<br>}<br>$ gcc one.c `pkg-config --cflags libdpdk --libs --static libdpdk`<br>$ sudo ./a.out <br>EAL: Detected CPU lcores: 64<br>EAL: Detected NUMA nodes: 4<br>EAL: Detected static linkage of DPDK<br>EAL: Multi-process socket /var/run/dpdk/rte/mp_socket<br>EAL: Selected IOVA mode 'VA'<br>EAL: No available 1048576 kB hugepages reported<br>EAL: VFIO support initialized<br>EAL: Using IOMMU type 1 (Type 1)<br>EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:c3:00.0 (socket 3)<br>set_mempolicy: Invalid argument<br>EAL: Releasing PCI mapped resource for 0000:c3:00.0<br>EAL: Calling pci_unmap_resource for 0000:c3:00.0 at 0x4102000000<br>EAL: Calling pci_unmap_resource for 0000:c3:00.0 at 0x4104000000<br>EAL: Requested device 0000:c3:00.0 cannot be used<br>EAL: Using IOMMU type 1 (Type 1)<br>EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:c3:00.1 (socket 3)<br>set_mempolicy: Invalid argument<br>EAL: Releasing PCI mapped resource for 0000:c3:00.1<br>EAL: Calling pci_unmap_resource for 0000:c3:00.1 at 0x4104010000<br>EAL: Calling pci_unmap_resource for 0000:c3:00.1 at 0x4106010000<br>EAL: Requested device 0000:c3:00.1 cannot be used<br>main 20<br>main 23 nb_ports: 0<br></div><div dir="ltr"><br></div><div dir="ltr"><div dir="ltr"><a class="gmail_plusreply" id="gmail-plusReplyChip-0">+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</a><br></div><div dir="ltr"><br></div><div>Please let me know if any more information is required.<br></div></div><div dir="ltr"><br><br>Thanks & Regards<br>--<br>Lokesh Chakka.<br></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 2, 2022 at 2:49 PM Van Haaren, Harry <<a href="mailto:harry.van.haaren@intel.com">harry.van.haaren@intel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div style="overflow-wrap: break-word;" lang="EN-GB"> <div class="gmail-m_2385346076083591834WordSection1"> <p class="MsoNormal"><font size="2" face="Calibri" color="#f6f6f6"><span style="font-size:11pt;color:black">[Top posting, as HTML email]<u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri" color="#f6f6f6"><span style="font-size:11pt;color:black"><u></u> <u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri" color="#f6f6f6"><span style="font-size:11pt;color:black">HI Chakka,<u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri" color="#f6f6f6"><span style="font-size:11pt;color:black"><u></u> <u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri" color="#f6f6f6"><span style="font-size:11pt;color:black">When linking against DPDK in Shared-library mode, the .so files to handle drivers are not automatically loaded;<u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri" color="#f6f6f6"><span style="font-size:11pt;color:black"><a href="https://doc.dpdk.org/guides-20.05/linux_gsg/linux_eal_parameters.html#linux-specific-eal-parameters" target="_blank">https://doc.dpdk.org/guides-20.05/linux_gsg/linux_eal_parameters.html#linux-specific-eal-parameters</a></span></font><span><u></u><u></u></span></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">Notice the -d flag in particular: "-d <path to shared object or directory>", when used and pointed to the correct .so for the ice pmd .so file,<u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">DPDK will be able to initialize the NIC.<u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">As a workaround/alternative, static-linking against DPDK will avoid the requirement to use "-d" EAL arg to load the PCI device .so drivers.<u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">Hope that helps, -Harry<u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri" color="#f6f6f6"><span style="font-size:11pt;color:black"><u></u> <u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri" color="#f6f6f6"><span style="font-size:11pt;color:black"><u></u> <u></u></span></font></p> <div style="border-color:currentcolor currentcolor currentcolor blue;border-style:none none none solid;border-width:medium medium medium 1.5pt;padding:0cm 0cm 0cm 4pt"> <div> <div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm"> <p class="MsoNormal"><b><font size="2" face="Calibri"><span style="font-size:11pt;font-weight:bold" lang="EN-US">From:</span></font></b><span lang="EN-US"> Lokesh Chakka <<a href="mailto:lvenkatakumarchakka@gmail.com" target="_blank">lvenkatakumarchakka@gmail.com</a>> <br> <b><span style="font-weight:bold">Sent:</span></b> Monday, August 1, 2022 9:20 AM<br> <b><span style="font-weight:bold">To:</span></b> Tkachuk, Georgii <<a href="mailto:georgii.tkachuk@intel.com" target="_blank">georgii.tkachuk@intel.com</a>><br> <b><span style="font-weight:bold">Cc:</span></b> users <<a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a>><br> <b><span style="font-weight:bold">Subject:</span></b> Re: intel 100 Gbps Network card shutting down after enabling dpdk<u></u><u></u></span></p> </div> </div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> <div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">After booting the server, the following are commands I have executed. Immediately after doing devbind -b, cards are going down. I am able to see that green leds are turning off.<u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">following is the two lines of code and output of code. number of ports are reporting as zero<u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> </div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<u></u><u></u></span></font></p> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">$ lspci | grep -i intel<br> c3:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)<br> c3:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)<br> <br> $ sudo dpdk-devbind.py -b vfio-pci c3:00.0 c3:00.1<br> $ sudo dpdk-devbind.py -s<br> <br> Network devices using DPDK-compatible driver<br> ============================================<br> 0000:c3:00.0 'Ethernet Controller E810-C for QSFP 1592' drv=vfio-pci unused=ice<br> 0000:c3:00.1 'Ethernet Controller E810-C for QSFP 1592' drv=vfio-pci unused=ice<br> <br> Network devices using kernel driver<br> ===================================<br> 0000:05:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=eno1 drv=tg3 unused=vfio-pci <br> 0000:05:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=eno2 drv=tg3 unused=vfio-pci <br> 0000:05:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=eno3 drv=tg3 unused=vfio-pci <br> 0000:05:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=eno4 drv=tg3 unused=vfio-pci <br> 0000:83:00.0 'BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet 1750' if=ens3f0np0 drv=bnxt_en unused=vfio-pci <br> 0000:83:00.1 'BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet 1750' if=ens3f1np1 drv=bnxt_en unused=vfio-pci <br> <br> No 'Baseband' devices detected<br> ==============================<br> <br> Crypto devices using kernel driver<br> ==================================<br> 0000:02:00.1 'Zeppelin Cryptographic Coprocessor NTBCCP 1468' drv=ccp unused=vfio-pci <br> 0000:03:00.2 'Family 17h (Models 00h-0fh) Platform Security Processor 1456' drv=ccp unused=vfio-pci <br> 0000:41:00.1 'Zeppelin Cryptographic Coprocessor NTBCCP 1468' drv=ccp unused=vfio-pci <br> 0000:42:00.2 'Family 17h (Models 00h-0fh) Platform Security Processor 1456' drv=ccp unused=vfio-pci <br> 0000:81:00.1 'Zeppelin Cryptographic Coprocessor NTBCCP 1468' drv=ccp unused=vfio-pci <br> 0000:82:00.2 'Family 17h (Models 00h-0fh) Platform Security Processor 1456' drv=ccp unused=vfio-pci <br> 0000:c1:00.1 'Zeppelin Cryptographic Coprocessor NTBCCP 1468' drv=ccp unused=vfio-pci <br> 0000:c2:00.2 'Family 17h (Models 00h-0fh) Platform Security Processor 1456' drv=ccp unused=vfio-pci <br> <br> No 'DMA' devices detected<br> =========================<br> <br> No 'Eventdev' devices detected<br> ==============================<br> <br> No 'Mempool' devices detected<br> =============================<br> <br> No 'Compress' devices detected<br> ==============================<br> <br> No 'Misc (rawdev)' devices detected<br> ===================================<br> <br> No 'Regex' devices detected<br> ===========================<br> <br> $ cat one.c <br> #include <inttypes.h><br> #include <rte_ethdev.h><br> <br> <br> int main(int argc, char *argv[])<br> {<br> struct rte_mempool *mbuf_pool;<br> unsigned nb_ports;<br> uint16_t portid;<br> <br> /* Initializion the Environment Abstraction Layer (EAL). 8< */<br> int ret = rte_eal_init(argc, argv);<br> if (ret < 0)<br> rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");<br> /* >8 End of initialization the Environment Abstraction Layer (EAL). */<br> <br> argc -= ret;<br> argv += ret;<br> <br> fprintf( stderr, "%s %d\n", __func__, __LINE__ );<br> /* Check that there is an even number of ports to send/receive on. */<br> nb_ports = rte_eth_dev_count_avail();<br> fprintf( stderr, "%s %d nb_ports: %u\n", __func__, __LINE__, nb_ports );<br> }<br> <br> $ gcc one.c `pkg-config --cflags libdpdk --libs libdpdk`<br> $ sudo ./a.out <br> EAL: Detected CPU lcores: 64<br> EAL: Detected NUMA nodes: 4<br> EAL: Detected shared linkage of DPDK<br> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket<br> EAL: Selected IOVA mode 'VA'<br> EAL: No available 1048576 kB hugepages reported<br> EAL: VFIO support initialized<br> EAL: Using IOMMU type 1 (Type 1)<br> EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:c3:00.0 (socket 3)<br> set_mempolicy: Invalid argument<br> EAL: Releasing PCI mapped resource for 0000:c3:00.0<br> EAL: Calling pci_unmap_resource for 0000:c3:00.0 at 0x4102000000<br> EAL: Calling pci_unmap_resource for 0000:c3:00.0 at 0x4104000000<br> EAL: Requested device 0000:c3:00.0 cannot be used<br> EAL: Using IOMMU type 1 (Type 1)<br> EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:c3:00.1 (socket 3)<br> set_mempolicy: Invalid argument<br> EAL: Releasing PCI mapped resource for 0000:c3:00.1<br> EAL: Calling pci_unmap_resource for 0000:c3:00.1 at 0x4104010000<br> EAL: Calling pci_unmap_resource for 0000:c3:00.1 at 0x4106010000<br> EAL: Requested device 0000:c3:00.1 cannot be used<br> TELEMETRY: No legacy callbacks, legacy socket not created<br> main 20<br> main 23 nb_ports: 0<u></u><u></u></span></font></p> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> </div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ <u></u><u></u></span></font></p> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><br> Thanks & Regards<br> --<br> Lokesh Chakka.<u></u><u></u></span></font></p> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> </div> </div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt"><u></u> <u></u></span></font></p> <div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt">On Mon, Aug 1, 2022 at 12:44 PM Tkachuk, Georgii <<a href="mailto:georgii.tkachuk@intel.com" target="_blank">georgii.tkachuk@intel.com</a>> wrote:<u></u><u></u></span></font></p> </div> <blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">Please check “sudo dpdk-devbind.py -s” to see if both ports are bound to vfio-pci <u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">And some questions: <u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">What DPDK app are you running? <u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">What are your EAL command line arguments? <u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">What do you mean by going down (as reported by what?) <u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">Thank you,<u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">Georgii<u></u><u></u></span></font></p> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> <div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm;border-color:currentcolor"> <p class="MsoNormal"><b><font size="2" face="Calibri"><span style="font-size:11pt;font-weight:bold" lang="EN-US">From:</span></font></b><span lang="EN-US"> Lokesh Chakka <<a href="mailto:lvenkatakumarchakka@gmail.com" target="_blank">lvenkatakumarchakka@gmail.com</a>> <br> <b><span style="font-weight:bold">Sent:</span></b> Monday, August 01, 2022 12:09 AM<br> <b><span style="font-weight:bold">To:</span></b> users <<a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a>><br> <b><span style="font-weight:bold">Subject:</span></b> intel 100 Gbps Network card shutting down after enabling dpdk<u></u><u></u></span></p> </div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> <div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">hello,<u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">Recently I have bought one intel network card <u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"><a href="https://ark.intel.com/content/www/us/en/ark/products/210969/intel-ethernet-network-adapter-e8102cqda2.html" target="_blank">https://ark.intel.com/content/www/us/en/ark/products/210969/intel-ethernet-network-adapter-e8102cqda2.html</a><u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">The moment I do devbind, cards are shutting down.<u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">$ lspci | grep -i intel<br> c3:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)<br> c3:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)<br> <br> $ sudo dpdk-devbind.py -b vfio-pci c3:00.0 c3:00.1<br> <br> After this, cards are going down. EAL is giving following error<u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">EAL: Requested device 0000:c3:00.1 cannot be used<br> EAL: Error - exiting with code: 1<br> Cause: Error: number of ports must be even<u></u><u></u></span></font></p> </div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"> <u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US">Did anyone face this issue or anyone know how to fix the issue?<u></u><u></u></span></font></p> </div> <div> <p class="MsoNormal"><font size="2" face="Calibri"><span style="font-size:11pt" lang="EN-US"><br> Thanks & Regards<br> --<br> Lokesh Chakka.<u></u><u></u></span></font></p> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> </div> </div> </div> </blockquote></div>