<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> Hi,<o:p></o:p> <o:p> </o:p> We run DPDK 19.11.3 with multi processes.<o:p></o:p> When using external memory for DPDK (instead of huge pages), we are unable to receive traffic in the secondary processes.<o:p></o:p> <o:p> </o:p> <ul style="margin-top:0in" type="disc"> <li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">We have external physical memory regions (excluded from Linux):<o:p></o:p></li></ul> <o:p> </o:p> [ULP-NG]# cat /proc/cmdline<o:p></o:p> console=ttyS0,19200 isolcpus=1-127 smp_affinity=1 root=/dev/ram0 rwmode=r net.ifnames=0 biosdevname=0 systemd.show_status=0<o:p></o:p> memmap=0x1000000!0x60000000 memmap=0x90000000!0x800000000 memmap=0x90000000!0x1800000000 quiet cloud-init=disabled<o:p></o:p> <o:p> </o:p> <ul style="margin-top:0in" type="disc"> <li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">We make all DPDK initializations in the primary process, including mapping the mentioned memory regions via “/dev/mem”.<o:p></o:p></li></ul> After these memory mapping, we register the external memory, as described in <a href="http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory"> http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory</a>, and allocate memory pools:<o:p></o:p> <o:p> </o:p> ...<o:p></o:p> /* Map physical to virtual */<o:p></o:p> int memFd = open("/dev/mem", O_RDWR);<o:p></o:p> extMemInfo[i].pageSize = pPagesInfo->maxPageSize;<o:p></o:p> extMemInfo[i].memRegSize = 0x0000000080000000;<o:p></o:p> extMemInfo[i].memRegAddr = mmap(NULL, extMemInfo[i].memRegSize, PROT_READ | PROT_WRITE,<o:p></o:p> MAP_SHARED, memFd, memRegPhysAddr[i]);<o:p></o:p> <o:p> </o:p> /* Create heap */<o:p></o:p> sprintf(extMemInfo[i].heapName, "extMemHeapSocket_%u", i);<o:p></o:p> rv = rte_malloc_heap_create(extMemInfo[i].heapName);<o:p></o:p> <o:p> </o:p> /* Save heap socket ID */<o:p></o:p> rv = rte_malloc_heap_get_socket(extMemInfo[i].heapName);<o:p></o:p> extMemInfo[i].socketId = rv;<o:p></o:p> <o:p> </o:p> /* Add memory region to heap */<o:p></o:p> rv = rte_malloc_heap_memory_add(extMemInfo[i].heapName, extMemInfo[i].memRegAddr,<o:p></o:p> extMemInfo[i].memRegSize, NULL, 0, extMemInfo[i].pageSize);<o:p></o:p> ...<o:p></o:p> /* Allocate memory pool */<o:p></o:p> memPool = rte_mempool_create(poolName, nbufs, DPDK_MBUF_SIZE, poolCacheSize,<o:p></o:p> sizeof(struct rte_pktmbuf_pool_private),<o:p></o:p> rte_pktmbuf_pool_init, NULL,<o:p></o:p> und_pktmbuf_init, NULL, extMemInfo[i].socketId, DPDK_NO_FLAGS);<o:p></o:p> ...<o:p></o:p> <o:p> </o:p> Please note, that during calls to “rte_malloc_heap_memory_add”, we see the following warnings:<o:p></o:p> <o:p> </o:p> EAL: WARNING! Base virtual address hint (0x2101005000 != 0x7ffff4627000) not respected!<o:p></o:p> EAL: This may cause issues with mapping memory into secondary processes<o:p></o:p> EAL: WARNING! Base virtual address hint (0x210100b000 != 0x7ffff4619000) not respected!<o:p></o:p> EAL: This may cause issues with mapping memory into secondary processes<o:p></o:p> <o:p> </o:p> And after executing “rte_mempool_create”, physical addresses of the allocated memory zones are bad:<o:p></o:p> <o:p> </o:p> [Jul 12 12:02:17] [1875] extMemConfig: heapName=extMemHeapSocket_0, socketId=256, memRegAddr=0x7fff58000000, memRegSize=2415919104, pageSize=2097152<o:p></o:p> [Jul 12 12:02:18] [1875] memZone: name=MP_ndPool_0, socket_id=256, vaddr=0x7fffe7e7bec0-0x7fffe7ffffc0, paddr=0xffffffffffffffff-0x1840ff, len=1589504, hugepage_sz=2MB<o:p></o:p> <o:p> </o:p> <ul style="margin-top:0in" type="disc"> <li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">In the next step, we spawn the secondary processes from the primary one, using fork().<o:p></o:p></li></ul> This way, all DPDK data and memory mappings are the same on all processes. For example:<o:p></o:p> <o:p> </o:p> Mapping in primary process:<o:p></o:p> cat /proc/1875/maps<o:p></o:p> ...<o:p></o:p> 7ffec8000000-7fff58000000 rw-s 1800000000 00:06 5 /dev/mem<o:p></o:p> 7fff58000000-7fffe8000000 rw-s 800000000 00:06 5 /dev/mem<o:p></o:p> ...<o:p></o:p> <o:p> </o:p> Mapping in secondary process:<o:p></o:p> cat /proc/2676/maps<o:p></o:p> ...<o:p></o:p> 7ffec8000000-7fff58000000 rw-s 1800000000 00:06 5 /dev/mem<o:p></o:p> 7fff58000000-7fffe8000000 rw-s 800000000 00:06 5 /dev/mem<o:p></o:p> ...<o:p></o:p> <o:p> </o:p> <ul style="margin-top:0in" type="disc"> <li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Unfortunately, when traffic is received by the secondary processes, we see the following printout on the primary process:<o:p></o:p></li></ul> <o:p> </o:p> i40e_dev_alarm_handler(): ICR0: malicious programming detected<o:p></o:p> <o:p> </o:p> <ul style="margin-top:0in" type="disc"> <li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Additionally, we saw no example of using external physical shared memory on secondary processes.<o:p></o:p></li></ul> DPDK test applications show usage of anonymous private memory on the primary process only.<o:p></o:p> <o:p> </o:p> What are we missing?<o:p></o:p> Can you please advise?<o:p></o:p> <o:p> </o:p> Thanks,<o:p></o:p> Asaf<o:p></o:p> <o:p> </o:p> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="656" style="width:492.0pt"> <tbody> <tr> <td width="191" style="width:143.05pt;border:none;border-right:solid #B8C8D8 1.0pt;padding:0in 0in 0in 0in"> <img border="0" width="177" height="32" style="width:1.8416in;height:.3333in" id="_x0000_i1030" src="https://www.radware.com/images/signature/sig_logo.jpg" alt="Radware"><o:p></o:p> </td> <td width="321" valign="top" style="width:240.9pt;border:none;border-right:solid #B8C8D8 1.0pt;padding:0in 12.0pt 0in 12.0pt"> Asaf Sinai ND SW Engineer Email: <a href="mailto:asafsi@radware.com">asafsi@radware.com</a> <o:p></o:p> </td> <td width="144" valign="top" style="width:108.05pt;padding:0in 0in 0in 12.0pt"> T:+972-72-3917050 M:+972-50-6518541 F:+972-3-6488662 <o:p></o:p> </td> </tr> <tr> <td style="padding:0in 0in 0in 0in"></td> <td style="padding:0in 0in 0in 0in"></td> <td style="padding:0in 0in 0in 0in"></td> </tr> <tr> <td colspan="3" style="padding:5.25pt 0in 5.25pt 0in"> <div style="border:solid #F1F0F0 4.5pt;padding:0in 0in 0in 0in"> <a href="https://www.radware.com/Resources/CampaignRedirector.html"><img border="0" width="640" height="76" style="width:6.6666in;height:.7916in" id="_x0000_i1029" src="https://d3vlk452dxjc85.cloudfront.net/cdn/images/signaturefooter.jpg" alt="Check out the latest and greatest from Radware"></a><o:p></o:p> </div> </td> </tr> <tr> <td colspan="3" style="padding:0in 0in 0in 0in"> <div style="border:none;border-top:solid #FEB317 3.0pt;padding:0in 0in 0in 0in;margin-top:1.5pt;margin-bottom:1.5pt"> <o:p></o:p> </div> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="656" style="width:492.0pt"> <tbody> <tr> <td valign="top" style="padding:0in 0in 0in 0in"> <a href="https://www.radware.com">www.radware.com</a><o:p></o:p> </td> <td style="padding:0in 0in 0in 0in"> <o:p></o:p> </td> <td valign="bottom" style="padding:0in 0in .75pt 0in"> <a href="https://blog.radware.com/"><img border="0" width="53" height="18" style="width:.55in;height:.1916in" id="_x0000_i1028" src="https://d3vlk452dxjc85.cloudfront.net/cdn/images/Community/radware_blog_img2.png" alt="Blog"></a> <a href="https://www.linkedin.com/companies/165642"><img border="0" width="19" height="18" style="width:.2in;height:.1916in" id="_x0000_i1027" src="https://www.radware.com/images/signature/linkedin.jpg"></a><a href="file://twitter.com/radware"><img border="0" width="19" height="18" style="width:.2in;height:.1916in" id="_x0000_i1026" src="https://www.radware.com/images/signature/twitter.jpg"></a> <a href="https://www.youtube.com/user/radwareinc"><img border="0" width="43" height="18" style="width:.45in;height:.1916in" id="_x0000_i1025" src="https://www.radware.com/images/signature/youtube.jpg" alt="youtube"></a><o:p></o:p> </td> </tr> </tbody> </table> </td> </tr> <tr> <td colspan="3" style="padding:15.0pt 0in 0in 0in"> Confidentiality note: This message, and any attachments to it, contains privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may not be disclosed, used, copied, or transmitted in any form or by any means without prior written permission from RADWARE. If you are not the intended recipient, delete the message and any attachments from your system without reading or copying it, and kindly notify the sender by e-mail. Thank you. <o:p></o:p> </td> </tr> <tr> <td colspan="3" style="padding:11.25pt 0in 0in 0in"> P Please consider your environmental responsibility before printing this e-mail <o:p></o:p> </td> </tr> </tbody> </table> <o:p> </o:p> </div> </body> </html>