<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-IN" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hi Dimitry<o:p></o:p> <o:p> </o:p> We know If the application does unmap, DPDK native heaps are not getting cleaned up in �native: use regular DPDK memory� memory type.<o:p></o:p> <o:p> </o:p> Can we get an API from DPDK community please to clean up the heaps given the VA and length to be freed ? Like below<o:p></o:p> <o:p> </o:p> <a href="https://doc.dpdk.org/api/rte__memory_8h.html#afb3c1f8be29fa15953cebdad3a9cd8eb">https://doc.dpdk.org/api/rte__memory_8h.html#afb3c1f8be29fa15953cebdad3a9cd8eb</a><o:p></o:p> <o:p> </o:p> <table class="MsoNormalTable" border="0" cellpadding="0" style="margin-left:4.5pt"> <tbody> <tr> <td style="padding:.75pt .75pt .75pt .75pt"></td> </tr> <tr> <td valign="bottom" style="padding:.75pt .75pt .75pt .75pt"> int rte_extmem_unregister<o:p></o:p> </td> <td valign="bottom" style="padding:.75pt .75pt .75pt .75pt"> (<o:p></o:p> </td> <td nowrap="" valign="bottom" style="padding:.75pt .75pt .75pt .75pt"> void * <o:p></o:p> </td> <td nowrap="" valign="bottom" style="padding:.75pt .75pt .75pt .75pt"> va_addr,<o:p></o:p> </td> </tr> <tr> <td valign="bottom" style="padding:.75pt .75pt .75pt .75pt"></td> <td valign="bottom" style="padding:.75pt .75pt .75pt .75pt"></td> <td nowrap="" valign="bottom" style="padding:.75pt .75pt .75pt .75pt"> size_t <o:p></o:p> </td> <td nowrap="" valign="bottom" style="padding:.75pt .75pt .75pt .75pt"> len <o:p></o:p> </td> </tr> <tr> <td valign="bottom" style="padding:.75pt .75pt .75pt .75pt"></td> <td valign="bottom" style="padding:.75pt .75pt .75pt .75pt"> )<o:p></o:p> </td> <td valign="bottom" style="padding:.75pt .75pt .75pt .75pt"></td> <td style="padding:.75pt .75pt .75pt .75pt"></td> </tr> </tbody> </table> <o:p> </o:p> Unregister external memory chunk with DPDK.<o:p></o:p> <div style="mso-element:para-border-div;border:none;border-left:solid #D0C000 3.0pt;padding:0cm 0cm 0cm 2.0pt;background:white;margin-left:0cm;margin-right:3.75pt"> Note<o:p></o:p> </div> <div style="mso-element:para-border-div;border:none;border-left:solid #D0C000 3.0pt;padding:0cm 0cm 0cm 2.0pt;background:white;margin-left:36.0pt;margin-right:3.75pt"> Using this API is mutually exclusive with rte_malloc family of API's.<o:p></o:p> This API will not perform any DMA unmapping. It is expected that user will do that themselves.<o:p></o:p> Before calling this function, all other processes must call rte_extmem_detach to detach from the memory area.<o:p></o:p> </div> Parameters<o:p></o:p> <table class="MsoNormalTable" border="0" cellpadding="0" style="margin-left:36.0pt"> <tbody> <tr> <td nowrap="" valign="top" style="padding:.75pt 24.0pt .75pt 0cm"> va_addr<o:p></o:p> </td> <td style="padding:.75pt .75pt .75pt 0cm"> Start of virtual area to unregister<o:p></o:p> </td> </tr> <tr> <td nowrap="" valign="top" style="padding:.75pt 24.0pt .75pt 0cm"> len<o:p></o:p> </td> <td style="padding:.75pt .75pt .75pt 0cm"> Length of virtual area to unregister<o:p></o:p> </td> </tr> </tbody> </table> Returns<o:p></o:p> <![if !supportLists]>� <![endif]>0 on success<o:p></o:p> <![if !supportLists]>� <![endif]>-1 in case of error, with rte_errno set to one of the following: EINVAL - one of the parameters was invalid ENOENT - memory chunk was not <o:p></o:p> If we get such API in native way , it will be good for us instead of changing the design and going all the way writing code for new memory type <o:p></o:p> <o:p> </o:p> Please suggest us can we get an API to clean up DPDK NATIVE POOLS if VA_ADDR and LEN is given ?<o:p></o:p> <o:p> </o:p> Thanks<o:p></o:p> Umakiran<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Umakiran Godavarthi (ugodavar) <ugodavar@cisco.com> Date: Monday, 26 September 2022 at 6:25 PM To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> Cc: anatoly.burakov@intel.com <anatoly.burakov@intel.com>, dev@dpdk.org <dev@dpdk.org>, stephen@networkplumber.org <stephen@networkplumber.org> Subject: Re: DPDK 19.11.5 Legacy Memory Design Query<o:p></o:p> </div> Thanks <a id="OWAAMF9BEDABBDB271A48B5873C869C70D9D0" href="mailto:dmitry.kozliuk@gmail.com"> @Dmitry Kozlyuk</a> for your suggestions<o:p></o:p> <o:p></o:p> I will try the following for DPDK pool creation <o:p></o:p> <o:p></o:p> My logic of calculating MBUF�s remains same<o:p></o:p> <o:p></o:p> Saw this code in DPDK testpmd where External heap memory is used<o:p></o:p> <o:p></o:p> case MP_ALLOC_XMEM_HUGE:<o:p></o:p> {<o:p></o:p> int heap_socket;<o:p></o:p> bool huge = mp_alloc_type == MP_ALLOC_XMEM_HUGE;<o:p></o:p> <o:p></o:p> if (setup_extmem(nb_mbuf, mbuf_seg_size, huge) < 0)<o:p></o:p> rte_exit(EXIT_FAILURE, "Could not create external memory\n");<o:p></o:p> <o:p></o:p> heap_socket =<o:p></o:p> rte_malloc_heap_get_socket(EXTMEM_HEAP_NAME);<o:p></o:p> if (heap_socket < 0)<o:p></o:p> rte_exit(EXIT_FAILURE, "Could not get external memory socket ID\n");<o:p></o:p> <o:p></o:p> TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",<o:p></o:p> rte_mbuf_best_mempool_ops());<o:p></o:p> rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,<o:p></o:p> mb_mempool_cache, 0, mbuf_seg_size,<o:p></o:p> heap_socket);<o:p></o:p> break;<o:p></o:p> }<o:p></o:p> <o:p></o:p> So I will do the same<o:p></o:p> <o:p></o:p> <ol style="margin-top:0cm" start="1" type="1"> <li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3">EAL Init<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3">Calculate MBUF we need for our application<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3">Then create pool using MP_ALLOC_XEM_HUGE<o:p></o:p></li></ol> <o:p></o:p> 1,2,3 should work right , that should avoid heap corruption issues right ?<o:p></o:p> <o:p></o:p> We have total 3 types pool creation<o:p></o:p> <o:p></o:p> /*<o:p></o:p> * Select mempool allocation type:<o:p></o:p> * - native: use regular DPDK memory<o:p></o:p> * - anon: use regular DPDK memory to create mempool, but populate using<o:p></o:p> * anonymous memory (may not be IOVA-contiguous)<o:p></o:p> * - xmem: use externally allocated hugepage memory<o:p></o:p> */<o:p></o:p> <o:p></o:p> Instead of Freeing unused virtual memory in DPDK native way, we just create a pool with type XMEM_HUGE and add page by page to it like testpmd code<o:p></o:p> <o:p></o:p> Please let me know 1,2,3 steps are good right, so we need to boot DPDK with socket mem<o:p></o:p> <o:p></o:p> --socket-mem 2048 so that DPDK takes only 1 page natively and boots up right ?<o:p></o:p> <o:p></o:p> Thanks<o:p></o:p> Umakiran<o:p></o:p> <o:p></o:p> <o:p></o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> Date: Friday, 23 September 2022 at 6:41 PM To: Umakiran Godavarthi (ugodavar) <ugodavar@cisco.com> Cc: anatoly.burakov@intel.com <anatoly.burakov@intel.com>, dev@dpdk.org <dev@dpdk.org>, stephen@networkplumber.org <stephen@networkplumber.org> Subject: Re: DPDK 19.11.5 Legacy Memory Design Query<o:p></o:p> </div> <div> 2022-09-23 12:12 (UTC+0000), Umakiran Godavarthi (ugodavar): > [Uma] : Yes I agree if free_hp = 400, nr_hp = 252, we are expecting DPDK takes only 252 and keep the remaining pages free in its heap. > As you have mentioned just boot DPDK with 1 page, and add pages we want later, is this the steps > > 1. NR_HP =1 , FREE_HP = 1 > 2. EAL INIT (DPDK boots up with 1 2 MB page) > 3. What is the API for adding later on pages ? (rte_extmem_*, can you please give the full API details and how to call it with arguments ) Guide: <a href="https://doc.dpdk.org/guides-19.11/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory"> https://doc.dpdk.org/guides-19.11/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory</a> I recommend reading the entire section about DPDK memory management since you're going to use an uncommon API and should understand what's going on. API (the linked function and those following it): <a href="http://doc.dpdk.org/api-19.11/rte__malloc_8h.html#a2295623c85ba41fe5bf7dce6bf0393d6"> http://doc.dpdk.org/api-19.11/rte__malloc_8h.html#a2295623c85ba41fe5bf7dce6bf0393d6</a> <a href="http://doc.dpdk.org/api-19.11/rte__memory_8h.html#a653510fb0c58bf63f54708677e3a2eba"> http://doc.dpdk.org/api-19.11/rte__memory_8h.html#a653510fb0c58bf63f54708677e3a2eba</a> > We can do 1,2,3 there is a problem once we reduce pages to 1 , kernel will free the huge pages totally > > So is there a way not to touch NR_HP, FREE_HP, and just pass arguments to boot DPDK with just 1 page ? Please let us know and later add pages we need to DPDK !! See --socket-mem EAL option: <a href="http://doc.dpdk.org/guides-19.11/linux_gsg/linux_eal_parameters.html#id3"> http://doc.dpdk.org/guides-19.11/linux_gsg/linux_eal_parameters.html#id3</a> > Why do you need legacy mode in the first place? > Looks like you're painfully trying to achieve the same result > that dynamic mode would give you automatically. > > [Uma] : Yes , we can�t avoid legacy memory design as secondary process mapped page by page to primary process , and physical addr space is same for both processes. We have to stick to legacy memory design only for now !! Sorry, I still don't understand. Virtual and physical addresses of DPDK memory are the same across processes in both legacy and dynamic memory mode.<o:p></o:p> </div> </div> </body> </html>