<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> <p class="MsoNormal"><span style="color:#1F497D">Memory allocated with ‘no-huge’ will <b>not</b> be physically contiguous. Whether it’s cacheable is, I assume, depending on your platform. I can’t comment on ARM platforms, so I’ll defer to Jerin et al. to comment on this </span><span style="font-family:"Segoe UI Emoji",sans-serif;color:#1F497D">😊</span><span style="color:#1F497D"><o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p> <div> <p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#1F497D">Anatoly<o:p></o:p></span></p> </div> <p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b>From:</b> Kinsella, Ray <ray.kinsella@intel.com> <br> <b>Sent:</b> Tuesday, August 9, 2022 12:44 PM<br> <b>To:</b> Nick Tian <nick.tian@longsailingsemi.com>; Burakov, Anatoly <anatoly.burakov@intel.com>; users@dpdk.org<br> <b>Cc:</b> Jason Liu <jason.liu@longsailingsemi.com>; Sunshine Qin <sunshine.qin@longsailingsemi.com>; Mediter Li <mediter.li@longsailingsemi.com>; jerinj@marvell.com<br> <b>Subject:</b> RE: <span style="font-family:"MS Gothic"">回复：回复：</span>RE: About memory coherency<o:p></o:p></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New"">Hi Nick,<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New""><o:p> </o:p></span></p> <p class="MsoNormal"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New"">Adding Jerin to direct your query. As I think you need some of the ARM guys to chime in here, coherency constraints are ARM is different to Intel. <o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New""><o:p> </o:p></span></p> <p class="MsoNormal"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New"">Ray K<o:p></o:p></span></p> <p class="MsoNormal"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New""><o:p> </o:p></span></p> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal" style="margin-left:.5in"><b>From:</b> Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com">nick.tian@longsailingsemi.com</a>> <br> <b>Sent:</b> Tuesday 9 August 2022 12:32<br> <b>To:</b> Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com">nick.tian@longsailingsemi.com</a>>; Burakov, Anatoly <<a href="mailto:anatoly.burakov@intel.com">anatoly.burakov@intel.com</a>>; Kinsella, Ray <<a href="mailto:ray.kinsella@intel.com">ray.kinsella@intel.com</a>>; <a href="mailto:users@dpdk.org">users@dpdk.org</a><br> <b>Cc:</b> Jason Liu <<a href="mailto:jason.liu@longsailingsemi.com">jason.liu@longsailingsemi.com</a>>; Sunshine Qin <<a href="mailto:sunshine.qin@longsailingsemi.com">sunshine.qin@longsailingsemi.com</a>>; Mediter Li <<a href="mailto:mediter.li@longsailingsemi.com">mediter.li@longsailingsemi.com</a>><br> <b>Subject:</b> <span style="font-family:"MS Gothic"">回复：回复：</span>RE: About memory coherency<o:p></o:p></p> </div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE"><o:p> </o:p></span></p> <div id="mailbox-conversation"> <div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Hi Anatoly<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Let me provide some detail about our user sce</span><span lang="EN-IE" style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;background:#333333"></span><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">nario.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">1.Our HW platform does not support IOMMU, it's a ARM A53 based platform.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">2.We try to use DPDK to accelerate the ethernet forwarding performance with our own HW accelerator(Let use HAC as abbreviate).<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">3.We configure DPDK as "no-huge", and try to use rte_malloc to allocate memory and pass the PHY address of this memory to HAC(write the PHY address to it's register).<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">4.When HAC receive the ethernet packet, it will parse the packet data and write the descriptor and payload to the memory which allocated in step 3.(The HAC know where to write since we already tell it the address with it's register).<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">5.Application use the virtual address of this memory to read the descriptor and payload to do something else.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">As far as you know, the memory which would be passed to HW should be "uncacheable" and "physically continous" to avoid the in-coherency issue and other issue.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">(Given 64 bytes start with Address1 has already in CPU L2 cache, in the meantime , HAC write 32 bytes from Address1, then in-coherency issue will happen. since if we read the content from Address1 with software, the software will get<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">the old copy from cache instead of the latest one in DDR).<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">So, I need to know the memory which allocated by rte_malloc( configure with no-huge) is </span><strong><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:#E2001C;background:white">UNCACHEABLE/Physically continous </span></strong><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">or not? <o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">We need it </span><strong><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:#E2001C">physically continous</span></strong><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> and </span><strong><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:#E2001C">UNCACHEABLE</span></strong><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">, otherwise it will not suitable for this user scenario.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <div id="mailbox-conversation"> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:7.5pt">------------------</span><span style="font-size:7.5pt;font-family:"MS Gothic"">原始</span><span style="font-size:7.5pt;font-family:"Microsoft JhengHei",sans-serif">邮件</span><span lang="EN-IE" style="font-size:7.5pt"> ------------------<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"Microsoft JhengHei",sans-serif">发件人</span></b><b><span lang="EN-IE">:</span></b><span lang="EN-IE">Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com">nick.tian@longsailingsemi.com</a>><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"Microsoft JhengHei",sans-serif">发送时间</span></b><b><span lang="EN-IE">:</span></b><span lang="EN-IE">08/09/22 17:41:35<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"MS Gothic"">收件人</span></b><b><span lang="EN-IE">:</span></b><span lang="EN-IE">Burakov, Anatoly <<a href="mailto:anatoly.burakov@intel.com">anatoly.burakov@intel.com</a>>, Kinsella, Ray <<a href="mailto:ray.kinsella@intel.com">ray.kinsella@intel.com</a>>, <a href="mailto:users@dpdk.org"> users@dpdk.org</a> <<a href="mailto:users@dpdk.org">users@dpdk.org</a>><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"MS Gothic"">抄送</span></b><b><span lang="EN-IE">:</span></b><span lang="EN-IE">Jason Liu <<a href="mailto:jason.liu@longsailingsemi.com">jason.liu@longsailingsemi.com</a>>, Sunshine Qin <<a href="mailto:sunshine.qin@longsailingsemi.com">sunshine.qin@longsailingsemi.com</a>>, Mediter Li <<a href="mailto:mediter.li@longsailingsemi.com">mediter.li@longsailingsemi.com</a>><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"MS Gothic"">主</span></b><b><span style="font-family:"Microsoft JhengHei",sans-serif">题</span></b><b><span lang="EN-IE">:</span></b><span style="font-family:"MS Gothic"">回复：</span><span lang="EN-IE">RE: About memory coherency<o:p></o:p></span></p> </div> </div> <div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div id="mailbox-conversation"> <div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Hi Burakov<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Thanks for your reply.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">BTW, about the memory reserved by <span style="background:#F9F9F9">memfd_create-->ftruncate-->mmap, </span><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:#F9F9F9">what on earth is the coherency between cache and DDR? In another word, is it cacheable?uncacheable?</span><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:#F9F9F9">Is it possible for application to pass this memory to a device with DMA controller(I mean pass the PHY addr coverted by rte_mem_virt2phy to DMA controller)?</span><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:#F9F9F9">If yes, how can we ensure the coherency between cache and DDR?</span><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">static int eal_legacy_hugepage_init(void)<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> memfd = memfd_create("nohuge", 0);<br> ...<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> fd = memfd;<br> flags = MAP_SHARED; //MAP_SHARED means </span><strong><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:#E2001C">UNCACHEABLE</span></strong><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">?<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <div id="mailbox-conversation"> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:7.5pt">------------------</span><span style="font-size:7.5pt;font-family:"MS Gothic"">原始</span><span style="font-size:7.5pt;font-family:"Microsoft JhengHei",sans-serif">邮件</span><span lang="EN-IE" style="font-size:7.5pt"> ------------------<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"Microsoft JhengHei",sans-serif">发件人</span></b><b><span lang="EN-IE">:</span></b><span lang="EN-IE">Burakov, Anatoly <<a href="mailto:anatoly.burakov@intel.com">anatoly.burakov@intel.com</a>><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"Microsoft JhengHei",sans-serif">发送时间</span></b><b><span lang="EN-IE">:</span></b><span lang="EN-IE">08/09/22 17:25:35<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"MS Gothic"">收件人</span></b><b><span lang="EN-IE">:</span></b><span lang="EN-IE">Kinsella, Ray <<a href="mailto:ray.kinsella@intel.com">ray.kinsella@intel.com</a>>, Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com">nick.tian@longsailingsemi.com</a>>, <a href="mailto:users@dpdk.org">users@dpdk.org</a> <<a href="mailto:users@dpdk.org">users@dpdk.org</a>><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:"MS Gothic"">主</span></b><b><span style="font-family:"Microsoft JhengHei",sans-serif">题</span></b><b><span lang="EN-IE">:</span></b><span lang="EN-IE">RE: About memory coherency<o:p></o:p></span></p> </div> </div> <div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="color:#1F497D">There are two different issues at play here.</span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="color:#1F497D"> </span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="color:#1F497D">The purpose of “no-huge” flag is to run DPDK without requiring hugepage memory. Originally, this has been done using an anonymous mmap() call – so, this memory was not using any fd’s at all. This presents a problem with vhost-user, because it relies on fd’s for its shared memory implementation. This is what memfd (a relatively recent addition to the kernel) is addressing – it’s enabling usage of vhost-user with no-huge because memfd actually does create an fd to back our memory.</span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="color:#1F497D"> </span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="color:#1F497D">That said, while description says “malloc”, it is<i>technically</i> incorrect because there’s no malloc involved in the process. The “malloc” term is simply shorthand for “use regular memory”, and should be understood in that context.</span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="color:#1F497D"> </span><span lang="EN-IE"><o:p></o:p></span></p> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="color:#1F497D">Thanks,</span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="color:#1F497D">Anatoly</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="color:#1F497D"> </span><span lang="EN-IE"><o:p></o:p></span></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal" style="margin-left:.5in"><b><span lang="EN-IE">From:</span></b><span lang="EN-IE"> Kinsella, Ray <<a href="mailto:ray.kinsella@intel.com">ray.kinsella@intel.com</a>> <br> <b>Sent:</b> Tuesday, August 9, 2022 10:04 AM<br> <b>To:</b> Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com">nick.tian@longsailingsemi.com</a>>; <a href="mailto:users@dpdk.org">users@dpdk.org</a><br> <b>Cc:</b> Burakov, Anatoly <<a href="mailto:anatoly.burakov@intel.com">anatoly.burakov@intel.com</a>><br> <b>Subject:</b> RE: About memory coherency<o:p></o:p></span></p> </div> </div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE"> <o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New""> </span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New"">I may be incorrect, but is it not simply the case, that when using the no-huge parameter that MAP_HUGETLB is omitted from flags?</span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New""> </span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New"">Ray K</span><span lang="EN-IE"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:12.0pt;font-family:"Courier New""> </span><span lang="EN-IE"><o:p></o:p></span></p> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal" style="margin-left:1.0in"><b><span lang="EN-IE">From:</span></b><span lang="EN-IE"> Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com" target="_blank">nick.tian@longsailingsemi.com</a>><br> <b>Sent:</b> Tuesday 9 August 2022 03:55<br> <b>To:</b> <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a><br> <b>Subject:</b> About memory coherency<o:p></o:p></span></p> </div> <p class="MsoNormal" style="margin-left:1.0in"><span lang="EN-IE"> <o:p></o:p></span></p> <div id="mailbox-conversation"> <div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div id="mailbox-conversation"> <div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Hi</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">I am confusing about the "no-huge" option of DPDK 21.11.</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">The dpdk usage said: --no-huge:Use malloc instead of hugetlbfs.</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">But when I check the EAL source code, I found some code piece like this:</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">It's look like "no-huge" option will lead dpdk use memfd_create-->ftruncate-->mmap to reserve memory</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">and then provide to application with rte_malloc.</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Am I right?</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">If so, what the "malloc" in "use malloc instead of hugelbfs" refer to?</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> </span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">EAL_memory.c</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">static int eal_legacy_hugepage_init(void){</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">....</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> if (internal_conf->no_hugetlbfs) {<br> ....<br> #ifdef MEMFD_SUPPORTED<br> /* create a memfd and store it in the segment fd table */<br> memfd = memfd_create("nohuge", 0);<br> ......</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> /* we got an fd - now resize it */<br> if (ftruncate(memfd, internal_conf->memory) < 0) {<br> .....</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> fd = memfd;<br> flags = MAP_SHARED; }<br> ....<br> prealloc_addr = msl->base_va;<br> addr = mmap(prealloc_addr, mem_sz, PROT_READ | PROT_WRITE,<br> <span style="background:yellow;mso-highlight:yellow">flags</span> | MAP_FIXED, fd, 0);</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">...</span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> </span><span lang="EN-IE"><o:p></o:p></span></p> </div> <div> <div id="mailbox-conversation"> <p class="MsoNormal" style="margin-left:.5in"><span lang="EN-IE" style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> </span><span lang="EN-IE"><o:p></o:p></span></p> </div> </div> </div> </div> </blockquote> </div> </div> </div> </blockquote> </div> </blockquote> </div> </blockquote> </div> </blockquote> </div> </div> </div> </body> </html>