<div id="mailbox-conversation"><div style="font-family:微软雅黑;font-size:14px;color:#000000"><div>Hi
<span style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 22.4px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">Anatoly</span></div><div><span style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 22.4px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">Let me provide some detail about our user sce<span style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 22.4px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important; background-color: rgb(51, 51, 51);"></span>nario.</span></div><div><span style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 22.4px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">1.Our HW platform does not support IOMMU, it's a ARM A53 based platform.</span></div><div><span style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 22.4px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">2.We try to use DPDK to accelerate the ethernet forwarding performance with our own HW accelerator(Let use HAC as abbreviate).</span></div><div><span style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 22.4px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">3.We configure DPDK as "no-huge", and try to use rte_malloc to allocate memory and pass the PHY address of this memory to HAC(write the PHY address to it's register).</span></div><div><span style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 22.4px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">4.When HAC receive the ethernet packet, it will parse the packet data and write the descriptor and payload to the memory which allocated in step 3.(The HAC know where to write since we already tell it the address with it's register).</span></div><div><span style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 22.4px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">5.Application use the virtual address of this memory to read the descriptor and payload to do something else.</span></div><div>As far as you know, the memory which would be passed to HW should be "uncacheable" and "physically continous" to avoid the in-coherency issue and other issue.</div><div>(Given 64 bytes start with Address1 has already in CPU L2 cache, in the meantime , HAC write 32 bytes from Address1, then in-coherency issue will happen. since if we read the content from Address1 with software, the software will get</div><div>the old copy from cache instead of the latest one in DDR).</div><div><br></div><div>So, I need to know the memory which allocated by rte_malloc(
<span style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 22.4px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">configure with</span>
no-huge) is <strong><span style="color: rgb(226, 0, 28); background-color: rgb(255, 255, 255);">UNCACHEABLE/Physically continous </span></strong>or not? </div><div>We need it <strong><span style="color: rgb(226, 0, 28);">physically continous</span></strong> and <strong><span style="color: rgb(226, 0, 28);">UNCACHEABLE</span></strong>, otherwise it will not suitable for this user scenario.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><div id="mailbox-conversation"><br></div></div><div><br class="ignorable"></div></div><blockquote><div><div style="font-size: 70%"> ------------------原始邮件 ------------------</div><div><b>发件人:</b>Nick Tian <nick.tian@longsailingsemi.com></div><div><b>发送时间:</b>08/09/22 17:41:35</div><div><b>收件人:</b>Burakov, Anatoly <anatoly.burakov@intel.com>, Kinsella, Ray <ray.kinsella@intel.com>, users@dpdk.org <users@dpdk.org></div><div><b>抄送:</b>Jason Liu <jason.liu@longsailingsemi.com>, Sunshine Qin <sunshine.qin@longsailingsemi.com>, Mediter Li <mediter.li@longsailingsemi.com></div><div><b>主题:</b>回复：RE: About memory coherency</div></div><div><blockquote class="gmail_quote"><div id="mailbox-conversation"><div style="font-family: 微软雅黑;font-size: 14.0px;color: #000000;"><div>Hi Burakov</div><div>Thanks for your reply.</div><div>BTW, about the memory reserved by <span style="color: #000000;font-family: microsoft yahei , sans-serif;font-size: 14.0px;font-style: normal;font-variant: normal;font-weight: normal;letter-spacing: normal;line-height: 22.4px;orphans: auto;text-align: start;text-indent: 0.0px;text-transform: none;white-space: normal;widows: 1;word-spacing: 0.0px;-webkit-text-stroke-width: 0.0px;display: inline;float: none;background-color: #f9f9f9;">memfd_create-->ftruncate-->mmap, </span></div><div><span style="color: #000000;font-family: microsoft yahei , sans-serif;font-size: 14.0px;font-style: normal;font-variant: normal;font-weight: normal;letter-spacing: normal;line-height: 22.4px;orphans: auto;text-align: start;text-indent: 0.0px;text-transform: none;white-space: normal;widows: 1;word-spacing: 0.0px;-webkit-text-stroke-width: 0.0px;display: inline;float: none;background-color: #f9f9f9;">what on earth is the coherency between cache and DDR? In another word, is it cacheable?uncacheable?</span></div><div><font face="microsoft yahei, sans-serif"><span style="background-color: #f9f9f9;">Is it possible for application to pass this memory to a device with DMA controller(I mean pass the PHY addr coverted by rte_mem_virt2phy to DMA controller)?</span></font></div><div><font face="microsoft yahei, sans-serif"><span style="background-color: #f9f9f9;">If yes, how can we ensure the coherency between cache and DDR?</span></font></div><div><br></div><div><br></div><div>static int eal_legacy_hugepage_init(void)</div><div> memfd = memfd_create("nohuge", 0);<br>...</div><div> fd = memfd;<br> flags = MAP_SHARED; //MAP_SHARED means <strong><span style="color: #e2001c;">UNCACHEABLE</span></strong>?</div><div><br></div><div><div id="mailbox-conversation"><br></div></div><div><br class="ignorable"></div></div><blockquote><div><div style="font-size: 70.0%;"> ------------------原始邮件 ------------------</div><div><b>发件人:</b>Burakov, Anatoly <anatoly.burakov@intel.com></div><div><b>发送时间:</b>08/09/22 17:25:35</div><div><b>收件人:</b>Kinsella, Ray <ray.kinsella@intel.com>, Nick Tian <nick.tian@longsailingsemi.com>, users@dpdk.org <users@dpdk.org></div><div><b>主题:</b>RE: About memory coherency</div></div><div><blockquote class="gmail_quote"><meta><meta><style>p.msonormal, li.msonormal, div.msonormal {margin: 0.0in;font-size: 11.0pt;font-family: calibri , sans-serif;}a:link, span.msohyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underline;}span.emailstyle20 {mso-style-type: personal-reply;font-family: calibri , sans-serif;color: #1f497d;}.msochpdefault {mso-style-type: export-only;font-size: 10.0pt;}div.wordsection1 {page: wordsection1;}</style><div class="WordSection1"><p class="MsoNormal"><span style="color: #1f497d;">There are two different issues at play here.</span></p><p class="MsoNormal"><span style="color: #1f497d;"> </span></p><p class="MsoNormal"><span style="color: #1f497d;">The purpose of “no-huge” flag is to run DPDK without requiring hugepage memory. Originally, this has been done using an anonymous mmap() call – so, this memory was not using any fd’s at all. This presents a problem with vhost-user, because it relies on fd’s for its shared memory implementation. This is what memfd (a relatively recent addition to the kernel) is addressing – it’s enabling usage of vhost-user with no-huge because memfd actually does create an fd to back our memory.</span></p><p class="MsoNormal"><span style="color: #1f497d;"> </span></p><p class="MsoNormal"><span style="color: #1f497d;">That said, while description says “malloc”, it is<i>technically</i> incorrect because there’s no malloc involved in the process. The “malloc” term is simply shorthand for “use regular memory”, and should be understood in that context.</span></p><p class="MsoNormal"><span style="color: #1f497d;"> </span></p><div><p class="MsoNormal"><span style="color: #1f497d;">Thanks,</span></p><p class="MsoNormal"><span style="color: #1f497d;">Anatoly</span></p></div><p class="MsoNormal"><span style="color: #1f497d;"> </span></p><div style="border: none;border-left: solid blue 1.5pt;padding: 0.0in 0.0in 0.0in 4.0pt;"><div><div style="border: none;border-top: solid #e1e1e1 1.0pt;padding: 3.0pt 0.0in 0.0in 0.0in;"><p class="MsoNormal"><b>From:</b> Kinsella, Ray <ray.kinsella@intel.com> <br><b>Sent:</b> Tuesday, August 9, 2022 10:04 AM<br><b>To:</b> Nick Tian <nick.tian@longsailingsemi.com>; users@dpdk.org<br><b>Cc:</b> Burakov, Anatoly <anatoly.burakov@intel.com><br><b>Subject:</b> RE: About memory coherency</p></div></div><p class="MsoNormal"> </p><p class="MsoNormal"><span style="font-size: 12.0pt;font-family: courier new;"> </span></p><p class="MsoNormal"><span style="font-size: 12.0pt;font-family: courier new;">I may be incorrect, but is it not simply the case, that when using the no-huge parameter that MAP_HUGETLB is omitted from flags?</span></p><p class="MsoNormal"><span style="font-size: 12.0pt;font-family: courier new;"> </span></p><p class="MsoNormal"><span style="font-size: 12.0pt;font-family: courier new;">Ray K</span></p><p class="MsoNormal"><span style="font-size: 12.0pt;font-family: courier new;"> </span></p><div style="border: none;border-top: solid #e1e1e1 1.0pt;padding: 3.0pt 0.0in 0.0in 0.0in;"><p class="MsoNormal" style="margin-left: 0.5in;"><b>From:</b> Nick Tian <<a href="mailto:nick.tian@longsailingsemi.com" target="_blank">nick.tian@longsailingsemi.com</a>><br><b>Sent:</b> Tuesday 9 August 2022 03:55<br><b>To:</b> <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a><br><b>Subject:</b> About memory coherency</p></div><p class="MsoNormal" style="margin-left: 0.5in;"><span> </span></p><div id="mailbox-conversation"><div><blockquote style="margin-top: 5.0pt;margin-bottom: 5.0pt;"><div id="mailbox-conversation"><div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">Hi</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">I am confusing about the "no-huge" option of DPDK 21.11.</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">The dpdk usage said: --no-huge:Use malloc instead of hugetlbfs.</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">But when I check the EAL source code, I found some code piece like this:</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">It's look like "no-huge" option will lead dpdk use memfd_create-->ftruncate-->mmap to reserve memory</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">and then provide to application with rte_malloc.</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">Am I right?</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">If so, what the "malloc" in "use malloc instead of hugelbfs" refer to?</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;"> </span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">EAL_memory.c</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">static int eal_legacy_hugepage_init(void){</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">....</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;"> if (internal_conf->no_hugetlbfs) {<br>....<br>#ifdef MEMFD_SUPPORTED<br> /* create a memfd and store it in the segment fd table */<br> memfd = memfd_create("nohuge", 0);<br>......</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;"> /* we got an fd - now resize it */<br> if (ftruncate(memfd, internal_conf->memory) < 0) {<br>.....</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;"> fd = memfd;<br> flags = MAP_SHARED; }<br>....<br> prealloc_addr = msl->base_va;<br> addr = mmap(prealloc_addr, mem_sz, PROT_READ | PROT_WRITE,<br> <span style="background: yellow;mso-highlight: yellow;">flags</span> | MAP_FIXED, fd, 0);</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;">...</span></p></div><div><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;"> </span></p></div><div><div id="mailbox-conversation"><p class="MsoNormal" style="margin-left: 0.5in;"><span style="font-size: 10.5pt;font-family: microsoft yahei , sans-serif;color: black;"> </span></p></div></div></div></div></blockquote></div></div></div></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div>