<div dir="ltr"><div>hi,</div><div><br></div><div>To add more information, the server I'm using has two CPU sockets and two NUMA nodes. One is numbered as Node 2 and other one as Node 6<br></div><div>one more observation is the following command is getting executed successfully.<br></div><div>$ sudo dpdk-hugepages.py -p 1G --setup 2G -n 2</div><div>also the following one <br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">$ sudo dpdk-hugepages.py -p 1G --setup 2G -n 6</div><div>After executing the first command, 1G huge pages are getting created. After executing the second command, huge pages under node 2 are getting deleted.<br></div><div dir="ltr"><br></div><div>Following is the output of dpdk-testpmd command</div><div><br></div><div><div>****************************************************************************************************<br></div><div>EAL: Detected CPU lcores: 128<br></div><div>EAL: Detected NUMA nodes: 8<br></div><div>EAL: Detected shared linkage of DPDK<br></div><div>EAL: Multi-process socket /var/run/dpdk/rte/mp_socket<br></div><div>EAL: Selected IOVA mode 'VA'<br></div><div>EAL: No free 1048576 kB hugepages reported on node 0<br></div><div>EAL: No free 1048576 kB hugepages reported on node 1<br></div><div>EAL: No free 1048576 kB hugepages reported on node 3<br></div><div>EAL: No free 1048576 kB hugepages reported on node 4<br></div><div>EAL: No free 1048576 kB hugepages reported on node 5<br></div><div>EAL: No free 1048576 kB hugepages reported on node 6<br></div><div>EAL: No free 1048576 kB hugepages reported on node 7<br></div><div>EAL: VFIO support initialized<br></div><div>set_mempolicy: Invalid argument<br></div><div>set_mempolicy: Invalid argument<br></div><div>set_mempolicy: Invalid argument<br></div><div>set_mempolicy: Invalid argument<br></div><div>EAL: Using IOMMU type 1 (Type 1)<br></div><div>set_mempolicy: Invalid argument<br></div><div>set_mempolicy: Invalid argument<br></div><div>EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:63:00.0 (socket 3)<br></div><div>set_mempolicy: Invalid argument<br></div><div>PANIC in eth_dev_shared_data_prepare():<br></div><div>Cannot allocate ethdev shared data<br></div><div>0: /lib/x86_64-linux-gnu/librte_eal.so.23 (rte_dump_stack+0x41) [7dbb0fe000b1]<br></div><div>1: /lib/x86_64-linux-gnu/librte_eal.so.23 (__rte_panic+0xc1) [7dbb0fde11c7]<br></div><div>2: /lib/x86_64-linux-gnu/librte_ethdev.so.23 (7dbb0fedb000+0x8b16) [7dbb0fee3b16]<br></div><div>3: /lib/x86_64-linux-gnu/librte_ethdev.so.23 (rte_eth_dev_allocate+0x31) [7dbb0feef971]<br></div><div>4: /usr/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ice.so.23 (7dbb0f70e000+0x67465) [7dbb0f775465]<br></div><div>5: /usr/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_pci.so.23 (7dbb0fc64000+0x4c76) [7dbb0fc68c76]<br></div><div>6: /usr/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_pci.so.23 (7dbb0fc64000+0x8af4) [7dbb0fc6caf4]<br></div><div>7: /lib/x86_64-linux-gnu/librte_eal.so.23 (rte_bus_probe+0x23) [7dbb0fdeeab3]<br></div><div>8: /lib/x86_64-linux-gnu/librte_eal.so.23 (7dbb0fdd4000+0x123bf) [7dbb0fde63bf]<br></div><div>9: dpdk-testpmd (5813e0022000+0x45150) [5813e0067150]<br></div><div>10: /lib/x86_64-linux-gnu/libc.so.6 (7dbb0ec00000+0x28150) [7dbb0ec28150]<br></div><div>11: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0x89) [7dbb0ec28209]<br></div><div>12: dpdk-testpmd (5813e0022000+0x48e55) [5813e006ae55]<br></div><div>Aborted<br></div><div><br></div><div>****************************************************************************************************<br></div><div><br></div><div></div><div><br></div></div><div dir="ltr"><br></div><div dir="ltr">Thanks & Regards<br>--<br>Lokesh Chakka.</div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 1, 2024 at 2:50 AM Lokesh Chakka <<a href="mailto:lvenkatakumarchakka@gmail.com">lvenkatakumarchakka@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>hi Stephen,</div><div><br></div><div>Thanks for the reply. Following is the observation...</div><div><br></div><div>*************************************************************</div><div>$ dpdk-hugepages.py -s<br>Node Pages Size Total<br>2 512 2Mb 1Gb<br>6 512 2Mb 1Gb<br><br>Hugepages mounted on /dev/hugepages /mnt/huge<br></div><div><br></div><div>$ sudo dpdk-hugepages.py -p 1G --setup 2G<br>Unable to set pages (0 instead of 2 in /sys/devices/system/node/node4/hugepages/hugepages-1048576kB/nr_hugepages).<br></div><div>*************************************************************<br></div><div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br><br>Regards<br>--<br>Lokesh Chakka.</div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 1, 2024 at 12:36 AM Stephen Hemminger <<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sun, 31 Mar 2024 16:28:19 +0530<br>
Lokesh Chakka <<a href="mailto:lvenkatakumarchakka@gmail.com" target="_blank">lvenkatakumarchakka@gmail.com</a>> wrote:<br>
<br>
> Hello,<br>
> <br>
> I've installed dpdk in Ubuntu 23.10 with the command "sudo apt -y install<br>
> dpdk*"<br>
> <br>
> added "nodev /mnt/huge hugetlbfs pagesize=1GB 0 0" in /etc/fstab<br>
> added "vm.nr_hugepages=1024" in /etc/sysctl.conf<br>
> <br>
> rebooted the machine and then did devbind using the following command:<br>
> <br>
> sudo modprobe vfio-pci && sudo dpdk-devbind.py --bind=vfio-pci 63:00.0<br>
> 63:00.1<br>
> <br>
> Huge page info is as follows :<br>
> <br>
> *************************************************<br>
> $ cat /proc/meminfo | grep Huge<br>
> AnonHugePages: 6144 kB<br>
> ShmemHugePages: 0 kB<br>
> FileHugePages: 0 kB<br>
> HugePages_Total: 1024<br>
> HugePages_Free: 1023<br>
> HugePages_Rsvd: 0<br>
> HugePages_Surp: 0<br>
> Hugepagesize: 2048 kB<br>
> Hugetlb: 2097152 kB<br>
> *************************************************<br>
<br>
Your hugepages are not setup correctly. The mount is for 1G pages<br>
and the sysctl entry makes 2M pages.<br>
<br>
Did you try using the dpdk-hugepages script?<br>
</blockquote></div>
</blockquote></div>