<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> <p class="MsoNormal"><span style="color:#993366">You should not use that memory’s real physical addresses, because we cannot guarantee that the kernel won’t change them under our feet. Please do not do that.<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#993366"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="color:#993366">However, if you use IOVA as VA mode (that is, if you have an IOMMU on your machine and you’re using VFIO to bind devices to DPDK), then no-huge memory <i>can</i> be used with IOMMU, because then the kernel/IOMMU takes care of all the VA to PA mappings. I honestly cannot answer the “cacheable” question, as it never comes up (at least on IA). What are you trying to do, and how is this relevant?<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#993366"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="color:#993366">In general, no-huge is meant to be a debug option, and is neither intended nor adequately tested for production workloads.<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#993366"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="color:#993366">Thanks,<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#993366">Anatoly<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#993366"><o:p> </o:p></span></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b>From:</b> Nick Tian <nick.tian@longsailingsemi.com> <br> <b>Sent:</b> Tuesday, August 9, 2022 10:42 AM<br> <b>To:</b> Burakov, Anatoly <anatoly.burakov@intel.com>; Kinsella, Ray <ray.kinsella@intel.com>; users@dpdk.org<br> <b>Cc:</b> Jason Liu <jason.liu@longsailingsemi.com>; Sunshine Qin <sunshine.qin@longsailingsemi.com>; Mediter Li <mediter.li@longsailingsemi.com><br> <b>Subject:</b> <span style="font-family:"MS Gothic"">回复：</span>RE: About memory coherency<o:p></o:p></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <div id="mailbox-conversation"> <div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Hi Burakov<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Thanks for your reply.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">BTW, about the memory reserved by <span style="background:#F9F9F9">memfd_create-->ftruncate-->mmap, </span><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:#F9F9F9">what on earth is the coherency between cache and DDR? In another word, is it cacheable?uncacheable?</span><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:#F9F9F9">Is it possible for application to pass this memory to a device with DMA controller(I mean pass the PHY addr coverted by rte_mem_virt2phy to DMA controller)?</span><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black;background:#F9F9F9">If yes, how can we ensure the coherency between cache and DDR?</span><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">static int eal_legacy_hugepage_init(void)<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> memfd = memfd_create("nohuge", 0);<br> ...<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> fd = memfd;<br> flags = MAP_SHARED; //MAP_SHARED means </span><strong><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:#E2001C">UNCACHEABLE</span></strong><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">?<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <div id="mailbox-conversation"> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> </div> <div> <p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"><o:p> </o:p></span></p> </div> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> <div> <p class="MsoNormal"><span style="font-size:7.5pt">------------------</span><span style="font-size:7.5pt;font-family:"MS Gothic"">原始</span><span style="font-size:7.5pt;font-family:"Microsoft JhengHei",sans-serif">邮件</span><span style="font-size:7.5pt"> ------------------<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><b><span style="font-family:"Microsoft JhengHei",sans-serif">发件人</span>:</b>Burakov, Anatoly <<a href="mailto:anatoly.burakov@intel.com">anatoly.burakov@intel.com</a>><o:p></o:p></p> </div> <div> <p class="MsoNormal"><b><span style="font-family:"Microsoft JhengHei",sans-serif">发送时间</span>:</b>08/09/22 17:25:35<o:p></o:p></p> </div> <div> <p class="MsoNormal"><b><span style="font-family:"MS Gothic"">收件人</span>:</b>Kinsella, Ray <<a href="mailto:ray.kinsella@intel.com">ray.kinsella@intel.com</a>>, Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com">nick.tian@longsailingsemi.com</a>>, <a href="mailto:users@dpdk.org">users@dpdk.org</a> <<a href="mailto:users@dpdk.org">users@dpdk.org</a>><o:p></o:p></p> </div> <div> <p class="MsoNormal"><b><span style="font-family:"MS Gothic"">主</span></b><b><span style="font-family:"Microsoft JhengHei",sans-serif">题</span>:</b>RE: About memory coherency<o:p></o:p></p> </div> </div> <div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <p class="MsoNormal"><span style="color:#1F497D">There are two different issues at play here.</span><o:p></o:p></p> <p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p> <p class="MsoNormal"><span style="color:#1F497D">The purpose of “no-huge” flag is to run DPDK without requiring hugepage memory. Originally, this has been done using an anonymous mmap() call – so, this memory was not using any fd’s at all. This presents a problem with vhost-user, because it relies on fd’s for its shared memory implementation. This is what memfd (a relatively recent addition to the kernel) is addressing – it’s enabling usage of vhost-user with no-huge because memfd actually does create an fd to back our memory.</span><o:p></o:p></p> <p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p> <p class="MsoNormal"><span style="color:#1F497D">That said, while description says “malloc”, it is<i>technically</i> incorrect because there’s no malloc involved in the process. The “malloc” term is simply shorthand for “use regular memory”, and should be understood in that context.</span><o:p></o:p></p> <p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p> <div> <p class="MsoNormal"><span style="color:#1F497D">Thanks,</span><o:p></o:p></p> <p class="MsoNormal"><span style="color:#1F497D">Anatoly</span><o:p></o:p></p> </div> <p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b>From:</b> Kinsella, Ray <<a href="mailto:ray.kinsella@intel.com">ray.kinsella@intel.com</a>> <br> <b>Sent:</b> Tuesday, August 9, 2022 10:04 AM<br> <b>To:</b> Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com">nick.tian@longsailingsemi.com</a>>; <a href="mailto:users@dpdk.org">users@dpdk.org</a><br> <b>Cc:</b> Burakov, Anatoly <<a href="mailto:anatoly.burakov@intel.com">anatoly.burakov@intel.com</a>><br> <b>Subject:</b> RE: About memory coherency<o:p></o:p></p> </div> </div> <p class="MsoNormal"> <o:p></o:p></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New"">I may be incorrect, but is it not simply the case, that when using the no-huge parameter that MAP_HUGETLB is omitted from flags?</span><o:p></o:p></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New"">Ray K</span><o:p></o:p></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New""> </span><o:p></o:p></p> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal" style="margin-left:.5in"><b>From:</b> Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com" target="_blank">nick.tian@longsailingsemi.com</a>><br> <b>Sent:</b> Tuesday 9 August 2022 03:55<br> <b>To:</b> <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a><br> <b>Subject:</b> About memory coherency<o:p></o:p></p> </div> <p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p> <div id="mailbox-conversation"> <div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div id="mailbox-conversation"> <div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Hi</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">I am confusing about the "no-huge" option of DPDK 21.11.</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">The dpdk usage said: --no-huge:Use malloc instead of hugetlbfs.</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">But when I check the EAL source code, I found some code piece like this:</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">It's look like "no-huge" option will lead dpdk use memfd_create-->ftruncate-->mmap to reserve memory</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">and then provide to application with rte_malloc.</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">Am I right?</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">If so, what the "malloc" in "use malloc instead of hugelbfs" refer to?</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> </span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">EAL_memory.c</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">static int eal_legacy_hugepage_init(void){</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">....</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> if (internal_conf->no_hugetlbfs) {<br> ....<br> #ifdef MEMFD_SUPPORTED<br> /* create a memfd and store it in the segment fd table */<br> memfd = memfd_create("nohuge", 0);<br> ......</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> /* we got an fd - now resize it */<br> if (ftruncate(memfd, internal_conf->memory) < 0) {<br> .....</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> fd = memfd;<br> flags = MAP_SHARED; }<br> ....<br> prealloc_addr = msl->base_va;<br> addr = mmap(prealloc_addr, mem_sz, PROT_READ | PROT_WRITE,<br> <span style="background:yellow;mso-highlight:yellow">flags</span> | MAP_FIXED, fd, 0);</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black">...</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> </span><o:p></o:p></p> </div> <div> <div id="mailbox-conversation"> <p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Microsoft YaHei",sans-serif;color:black"> </span><o:p></o:p></p> </div> </div> </div> </div> </blockquote> </div> </div> </div> </blockquote> </div> </blockquote> </div> </div> </div> </body> </html>