<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word"> <div class="WordSection1"> Hi Dmitry,<o:p></o:p> I hope you don’t mind if I reach out to you for hugepage memory mapping to memseg list issue that intermittently occurs.<o:p></o:p> <o:p> </o:p> We are seeing on occasion the DPDK allocation of hugepages fail.<o:p></o:p> DPDK version 22.11.2<o:p></o:p> Oracle 91 OS with kernel 5.14.0-284<o:p></o:p> The VM is configured with 32GB memory and 8 vCPU cores.<o:p></o:p> Setup for 2 x 1GB = 2GB hugepage total<o:p></o:p> We dynamically allocate hugepages before our application starts, is not done in grub but done in a bash script.<o:p></o:p> <o:p> </o:p> I turned on EAL debug in our application, which shows debug messages during EAL init.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Enable dpdk log EAL in nsprobe.<o:p></o:p> EAL: lib.eal log level changed from info to debug<o:p></o:p> EAL: Detected lcore 0 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 1 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 2 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 3 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 4 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 5 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 6 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 7 as core 0 on socket 0<o:p></o:p> EAL: Maximum logical cores by configuration: 128<o:p></o:p> EAL: Detected CPU lcores: 8<o:p></o:p> EAL: Detected NUMA nodes: 1<o:p></o:p> EAL: Checking presence of .so 'librte_eal.so.23.0'<o:p></o:p> EAL: Checking presence of .so 'librte_eal.so.23'<o:p></o:p> EAL: Checking presence of .so 'librte_eal.so'<o:p></o:p> EAL: Detected static linkage of DPDK<o:p></o:p> EAL: Ask a virtual area of 0x2000 bytes<o:p></o:p> EAL: Virtual area found at 0x100000000 (size = 0x2000)<o:p></o:p> [New Thread 0x7fed931ff640 (LWP 287600)]<o:p></o:p> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket<o:p></o:p> [New Thread 0x7fed929fe640 (LWP 287601)]<o:p></o:p> EAL: PCI driver net_iavf for device 0000:00:05.0 wants IOVA as 'PA'<o:p></o:p> EAL: PCI driver net_ice_dcf for device 0000:00:05.0 wants IOVA as 'PA'<o:p></o:p> EAL: PCI driver net_iavf for device 0000:00:06.0 wants IOVA as 'PA'<o:p></o:p> EAL: PCI driver net_ice_dcf for device 0000:00:06.0 wants IOVA as 'PA'<o:p></o:p> EAL: Bus pci wants IOVA as 'PA'<o:p></o:p> EAL: Bus vdev wants IOVA as 'DC'<o:p></o:p> EAL: Selected IOVA mode 'PA'<o:p></o:p> EAL: Probing VFIO support...<o:p></o:p> EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)<o:p></o:p> EAL: VFIO modules not loaded, skipping VFIO support...<o:p></o:p> EAL: Ask a virtual area of 0x2e000 bytes<o:p></o:p> EAL: Virtual area found at 0x100002000 (size = 0x2e000)<o:p></o:p> EAL: Setting up physically contiguous memory...<o:p></o:p> EAL: Setting maximum number of open files to 1024<o:p></o:p> EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824<o:p></o:p> EAL: Detected memory type: socket_id:0 hugepage_sz:2097152<o:p></o:p> EAL: Creating 1 segment lists: n_segs:2 socket_id:0 hugepage_sz:1073741824<o:p></o:p> EAL: Ask a virtual area of 0x1000 bytes<o:p></o:p> EAL: Virtual area found at 0x100030000 (size = 0x1000)<o:p></o:p> EAL: Memseg list allocated at socket 0, page size 0x100000kB<o:p></o:p> EAL: Ask a virtual area of 0x80000000 bytes<o:p></o:p> EAL: Virtual area found at 0x140000000 (size = 0x80000000)<o:p></o:p> EAL: VA reserved for memseg list at 0x140000000, size 80000000<o:p></o:p> EAL: Creating 1 segment lists: n_segs:1024 socket_id:0 hugepage_sz:2097152<o:p></o:p> EAL: Ask a virtual area of 0xd000 bytes<o:p></o:p> EAL: Virtual area found at 0x1c0000000 (size = 0xd000)<o:p></o:p> EAL: Memseg list allocated at socket 0, page size 0x800kB<o:p></o:p> EAL: Ask a virtual area of 0x80000000 bytes<o:p></o:p> EAL: Virtual area found at 0x1c0200000 (size = 0x80000000)<o:p></o:p> EAL: VA reserved for memseg list at 0x1c0200000, size 80000000<o:p></o:p> EAL: Trying to obtain current memory policy.<o:p></o:p> EAL: Setting policy MPOL_PREFERRED for socket 0<o:p></o:p> EAL: Setting policy MPOL_PREFERRED for socket 0<o:p></o:p> EAL: Restoring previous memory policy: 0<o:p></o:p> EAL: Hugepage /mnt/huge/rtemap_1 is on socket 0<o:p></o:p> EAL: Hugepage /mnt/huge/rtemap_0 is on socket 0<o:p></o:p> EAL: Requesting 2 pages of size 1024MB from socket 0 <<<< Same on good and bad<o:p></o:p> EAL: Attempting to map 1024M on socket 0 <<<< here, on good VM it states Attempting to map 2048M on socket 0, we have one numa node or 1 socket.<o:p></o:p> EAL: Allocated 1024M on socket 0 <<<< here, it allocated the 1024M on socket 0.<o:p></o:p> EAL: Attempting to map 1024M on socket 0 <<<< here, attempts to map last 1G to socket 0.<o:p></o:p> EAL: Could not find space for memseg. Please increase 1024 and/or 2048 in configuration. <<<<o:p></o:p> EAL: Couldn't remap hugepage files into memseg lists <<<< <o:p></o:p> EAL: FATAL: Cannot init memory<o:p></o:p> EAL: Cannot init memory<o:p></o:p> <o:p> </o:p> <o:p> </o:p> //good<o:p></o:p> EAL: Hugepage /mnt/huge/rtemap_1 is on socket 0<o:p></o:p> EAL: Hugepage /mnt/huge/rtemap_0 is on socket 0<o:p></o:p> EAL: Requesting 2 pages of size 1024MB from socket 0<o:p></o:p> EAL: Attempting to map 2048M on socket 0<o:p></o:p> EAL: Allocated 2048M on socket 0<o:p></o:p> EAL: Added 2048M to heap on socket 0<o:p></o:p> <o:p> </o:p> Could it be that the hugpages are not contiguous and reboot clears this issue, not able to confirm.<o:p></o:p> I tried rebooting the VM 10 times and could not get it to fail.<o:p></o:p> Tried multiple VMs and sometimes fails.<o:p></o:p> Seen on VMWare VM and openStack VMs.<o:p></o:p> <o:p> </o:p> Few months back you helped me reduce the VIRT memory of our application.<o:p></o:p> <o:p> </o:p> I added the following before building the dpdk static libraries that are used in our application build.<o:p></o:p> <o:p> </o:p> #define DPDK_REDUCE_VIRT_8G // is used to select the reduced MSL, etc reductions.<o:p></o:p> <o:p> </o:p> #if defined(DPDK_ORIGINAL) // original, VIRT: 36.6 GB<o:p></o:p> #define RTE_MAX_MEMSEG_LISTS 128<o:p></o:p> #define RTE_MAX_MEMSEG_PER_LIST 8192<o:p></o:p> #define RTE_MAX_MEM_MB_PER_LIST 32768<o:p></o:p> #define RTE_MAX_MEMSEG_PER_TYPE 32768<o:p></o:p> #define RTE_MAX_MEM_MB_PER_TYPE 65536<o:p></o:p> #endif<o:p></o:p> <o:p> </o:p> #if defined(DPDK_REDUCE_VIRT_8G) // VIRT: 5.9 GB<o:p></o:p> #define RTE_MAX_MEMSEG_LISTS 2<o:p></o:p> #define RTE_MAX_MEMSEG_PER_LIST 1024<o:p></o:p> #define RTE_MAX_MEM_MB_PER_LIST 2048<o:p></o:p> #define RTE_MAX_MEMSEG_PER_TYPE 1024<o:p></o:p> #defin<o:p></o:p> e RTE_MAX_MEM_MB_PER_TYPE 2048<o:p></o:p> #endif<o:p></o:p> <o:p> </o:p> We provide to rte_eal_init() the following arguments:<o:p></o:p> ‘app_name, -c0x2, -n4, --socket-mem=2048, --legacy-mem, --no-telemetry’<o:p></o:p> <o:p> </o:p> What do you suggest to eliminate this intermittent map to memseg list issue?<o:p></o:p> <o:p> </o:p> Thanks,<o:p></o:p> Ed<o:p></o:p> </div> </body> </html>