<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-IN" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> Hi Anatoly/DPDK-Developers<o:p></o:p> <o:p> </o:p> I am working on DPDK 19.11.5 Legacy Memory design and have a query about how to boot up in Legacy memory mode.<o:p></o:p> <o:p> </o:p> <ol style="margin-top:0cm" start="1" type="1"> <li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1"> Linux kernel boots up with huge pages (‘N’) and free huge pages (‘N’) initially <o:p></o:p></li></ol> <o:p></o:p> <ol style="margin-top:0cm" start="2" type="1"> <li class="MsoNormal" style="color:black;mso-list:l4 level1 lfo2">We calculate huge pages we need for data path (Driver need buffers for all queues) by sorting the memory fragments 2MB huge pages fragments and find we need ‘X’ pages , so we go to kernel and set SYSFS attribute nr_hugepages (/proc/sys/vm/nr_hugepages) to X pages. We also store the sorted physical memory fragments as POOL_0, POOL_1, POOL_2….etc. (Just for pool count purpose enough for all ports and queues to initialize)<o:p></o:p></li></ol> <o:p></o:p> For example, if host has memory pattern huge pages like this for total 500 we get in step 1 kernel reservation. <o:p></o:p> <o:p></o:p> 250, 90, 80, 70 , 10 -> Sum is 500 (N pages)<o:p></o:p> <o:p></o:p> We need only 350 pages (350 based on no of ports, queues dpdk application needs)<o:p></o:p> <o:p></o:p> So we need 250, 90, 10.<o:p></o:p> <o:p></o:p> So total 3 pools POOL_0 -> 250 pages, POOL_1 -> 90, POOL_2 -> 10 pages<o:p></o:p> <o:p></o:p> <ol style="margin-top:0cm" start="3" type="1"> <li class="MsoNormal" style="color:black;mso-list:l5 level1 lfo3">We boot up DPDK by RTE_EAL_INIT <o:p></o:p></li></ol> <o:p></o:p> <ol style="margin-top:0cm" start="4" type="1"> <li class="MsoNormal" style="color:black;mso-list:l1 level1 lfo4">Then we go to DPDK Memory segment list walkthrough and for each FBARRAY , we find the used pages by DPDK and unmap the remaining pages by below code (Idea is to free the huge pages taken by DPDK process virtual memory) -> Free_HP will be 0 then, as X pages are used by DPDK and all unnecessary pages are freed in this step)<o:p></o:p></li></ol> Sample Code of 4 :<o:p></o:p> <o:p></o:p> rte_memseg_list_walk_thread_unsafe(dpdk_find_and_free_unused, NULL); ->DPDK_FIND_AND_FREE_UNUSED is called for each Memory segment list (FBARRAY pointer is derived from MSL like below)<o:p></o:p> <o:p></o:p> dpdk_find_and_free_unused(const struct rte_memseg_list *msl,<o:p></o:p> void *arg UNUSED)<o:p></o:p> {<o:p></o:p> Int ms_idx;<o:p></o:p> arr = (struct rte_fbarray *) &msl->memseg_arr;<o:p></o:p> <o:p></o:p> /*<o:p></o:p> * use size of 2 instead of 1 to find the next free slot but <o:p></o:p> * not hole.<o:p></o:p> */<o:p></o:p> ms_idx = rte_fbarray_find_next_n_free(arr, 0, 2); <o:p></o:p> <o:p></o:p> if (ms_idx >= 0) {<o:p></o:p> addr = RTE_PTR_ADD(msl->base_va, ms_idx * msl->page_sz);<o:p></o:p> munmap(addr, RTE_PTR_DIFF(RTE_PTR_ADD(msl->base_va, msl->len), addr));<o:p></o:p> }<o:p></o:p> }<o:p></o:p> <o:p></o:p> <ol style="margin-top:0cm" start="5" type="1"> <li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo5">With NR_PAGES As ‘X’ and FREE_PAGES as 0, we create MBUF pools using RTE API and we face crash (We take care ‘X” pages has multi pools based on memory fragmentation given to the primary process, so pool by pool we have confidence that DPDK should find physical memory segment contiguous and allocate successfully)<o:p></o:p></li></ol> <o:p></o:p> struct rte_mempool *pool;<o:p></o:p> <o:p></o:p> pool = rte_pktmbuf_pool_create(name, num_mbufs,<o:p></o:p> RTE_MIN(num_mbufs/4, MBUF_CACHE_SIZE),<o:p></o:p> MBUF_PRIV_SIZE,<o:p></o:p> frame_len + RTE_PKTMBUF_HEADROOM,<o:p></o:p> rte_socket_id()<o:p></o:p> <o:p></o:p> <o:p></o:p> <ol style="margin-top:0cm" start="6" type="1"> <li class="MsoNormal" style="color:black;mso-list:l2 level1 lfo6">Sometimes randomly we face a crash during pool create in Step 5 for each POOL stored in Step 2 process for all ports and queues initialize later on<o:p></o:p></li></ol> <o:p></o:p> DPDK EAL core comes with BT like this<o:p></o:p> <o:p></o:p> #6 sigcrash (signo=11, info=0x7fff1c1867f0, ctx=0x7fff1c1866c0)<o:p></o:p> #7 <o:p></o:p> #8 malloc_elem_can_hold () from ./usr/lib64/dpdk-19/librte_eal.so.20.0<o:p></o:p> #9 find_suitable_element () from ./usr/lib64/dpdk-19/librte_eal.so.20.0<o:p></o:p> #10 malloc_heap_alloc () from ./usr/lib64/dpdk-19/librte_eal.so.20.0<o:p></o:p> #11 rte_malloc_socket () from ./usr/lib64/dpdk-19/librte_eal.so.20.0<o:p></o:p> #12 rte_mempool_create_empty () from ./usr/lib64/dpdk-19/librte_mempool.so.20.0<o:p></o:p> #13 rte_pktmbuf_pool_create_by_ops () from ./usr/lib64/dpdk-19/librte_mbuf.so.20.0<o:p></o:p> #14 rte_pktmbuf_pool_create () from ./usr/lib64/dpdk-19/librte_mbuf.so.20.0<o:p></o:p> #15 dpdk_create_mbuf_pool (mem_chunk_tbl=0x555d556de8e0 , num_mbufs=46080, frame_len=2048, name=0x7fff1c1873c0 "DPDK_POOL_0") <o:p></o:p> <o:p></o:p> We see find suitable element does not able to find a suitable element in DPDK memory segment lists it searches for HEAP ALLOC and returns NULL and NULL dereference crashes boot up process<o:p></o:p> <o:p></o:p> Please let me know any comments on boot up process for 1-6 and any reason behind the crash ?<o:p></o:p> <o:p></o:p> We are suspecting Step 4 where FBARRAY unused pages freeing at last should free the least contiguous memory segments right ? (munmap after finding 2 free pages , entire length we unmap in step 4 to free virtual memory)<o:p></o:p> <o:p> </o:p> Please let me know thoughts on FBARRAY design, is it expected to map the most contiguous…..least contiguous in a virtual address space right ?<o:p></o:p> <o:p> </o:p> So our most contiguous segments in Step 2 is safe even after Step 3, Step 4 we believe. Please correct my understanding if anything wrong.<o:p></o:p> <o:p></o:p> <o:p></o:p> Thanks<o:p></o:p> Umakiran<o:p></o:p> <o:p> </o:p> </div> </body> </html>