<div dir="ltr">I have tested the MAP_LOCKED, it doesn't help in this case. I do intend to report to the kernel but was wondering if others have hit upon this first.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 30, 2023 at 4:35 AM Stephen Hemminger <<a href="mailto:stephen@networkplumber.org">stephen@networkplumber.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sun, 28 May 2023 23:07:40 +0300<br> Baruch Even <<a href="mailto:baruch@weka.io" target="_blank">baruch@weka.io</a>> wrote:<br> <br> > Hi,<br> > <br> > We found an issue with newer kernels (5.13+) that are found on newer OSes<br> > (Ubuntu22, Rocky9, Ubuntu20 with kernel 5.15) where a 2M page that was<br> > allocated for DPDK was migrated (moved into another physical page) when a<br> > 1G page was allocated.<br> > <br> > From our reading of the kernel commits this started with commit<br> > ae37c7ff79f1f030e28ec76c46ee032f8fd07607<br> > mm: make alloc_contig_range handle in-use hugetlb pages<br> > <br> > This caused what looked like memory corruptions to us and cases where the<br> > rings were moved from their physical location and communication was no<br> > longer possible.<br> > <br> > I wanted to ask if anyone else hit this issue and what mitigations are<br> > available?<br> > <br> > We are currently looking at using a kernel driver to pin the pages but I<br> > expect that this issue will affect others and that a more general approach<br> > is needed.<br> > <br> > Thanks,<br> > Baruch<br> > <br> <br> Fix might be as simple as asking kernel to lock the mmap().<br> <br> diff --git a/lib/eal/linux/eal_hugepage_info.c b/lib/eal/linux/eal_hugepage_info.c<br> index 581d9dfc91eb..989c69387233 100644<br> --- a/lib/eal/linux/eal_hugepage_info.c<br> +++ b/lib/eal/linux/eal_hugepage_info.c<br> @@ -48,7 +48,8 @@ map_shared_memory(const char *filename, const size_t mem_size, int flags)<br> return NULL;<br> }<br> retval = mmap(NULL, mem_size, PROT_READ | PROT_WRITE,<br> - MAP_SHARED, fd, 0);<br> + MAP_SHARED_VALIDATE | MAP_LOCKED, fd, 0);<br> +<br> close(fd);<br> return retval == MAP_FAILED ? NULL : retval;<br> }<br> </blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div><table cellspacing="0" cellpadding="0" border="0"><tbody><tr><td style="font-size:0px;height:10px" height="10"></td></tr><tr><td style="font-size:0px;height:10px" height="10"></td></tr><tr><td><table style="width:510px" width="510" cellspacing="0" cellpadding="0" border="0"><tbody><tr><td><table cellspacing="0" cellpadding="0" border="0"></table></td></tr><tr><td style="padding-top:10px"> <div> <div style="max-width:510px"> <table style="width:510px" width="510" cellspacing="0" cellpadding="0" border="0"> <tbody><tr><td style="vertical-align:top;padding-right:12px;width:113px" width="113.0" valign="top"> <table cellspacing="0" cellpadding="0" border="0"> <tbody><tr> <td><img src="https://d4d8xd20er8lg.cloudfront.net/996a286d-104f-4862-86c3-38e6ff576b6b/uWcoN7kq5T52WAzl4ESe1yVt5JWBh1at.png" style="width: 113px; height: 82px; border-radius: 5px;" width="113.0" height="82.0"> </td> </tr> </tbody></table> </td><td> <table style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody><tr> <td style="padding-bottom:6px;border-bottom:1px solid rgb(172,172,172);font-size:16px;line-height:24px"> <span style="color:rgb(119,61,190);padding:0px;margin:0px;font-weight:bold;font-family:Arial;font-size:18px">Baruch Even</span><br> <span style="font-family:Arial">Platform Technical Lead, </span> <span style="color:rgb(119,61,190);font-family:Arial">WEKA</span> </td> </tr> <tr> <td style="padding-top:6px;padding-bottom:7px;line-height:16px"> <table style="color:rgb(68,68,68);padding-left:2px;font-weight:normal;width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody><tr> <td style="font-size:16px;line-height:24px"> <span style="color:rgb(119,61,190);font-weight:bold;font-family:Arial">E </span><a href="mailto:baruch@weka.io" style="text-decoration:none;color:rgb(78,75,76);font-family:Arial" target="_blank">baruch@weka.io</a><i style="color:rgb(255,255,255)"> </i><span style="color:rgb(119,61,190);font-weight:bold;font-family:Arial">W </span><a href="https://www.weka.io?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature" style="text-decoration:none;color:rgb(78,75,76);font-family:Arial" target="_blank">www.weka.io</a><i style="color:rgb(255,255,255)"> </i><span style="color:rgb(119,61,190);font-weight:bold;font-family:Arial"> </span><span style="color:rgb(78,75,76);font-family:Arial"></span><i style="color:rgb(255,255,255)"> </i> </td> </tr> </tbody></table> </td> </tr> <tr> <td> <table style="color:rgb(78,75,76);padding-left:2px;font-weight:bold;width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody><tr> <td><table cellspacing="0" cellpadding="0" border="0"><tbody><tr></tr></tbody></table></td> </tr> </tbody></table> </td> </tr> <tr> <td style="padding-top:6px"> </td> </tr> </tbody></table> </td> </tr> </tbody></table> </div></div></td></tr><tr><td><table cellspacing="0" cellpadding="0" border="0"><tbody><tr><td style="padding-top:8px;padding-right:8px"><div style="width:100%" width="100%"> <a href="https://www.weka.io/lp/weka-named-a-2023-customers-choice-by-gartner-peer-insights/?utm_source=signature&utm_medium=email" target="_blank"> <img src="https://d4d8xd20er8lg.cloudfront.net/7ff3fdb3-08f5-45bc-9e9f-af4382dbb940/u9B16WGa53zbO0yvcA1FtyAeRH14RmVD.png" style="width: 319px; height: 83px;" width="319" height="83"> </a></div></td></tr></tbody></table></td></tr></tbody></table></td></tr><tr><td><img style="display: none;" src="https://tracy.srv.wisestamp.com/px/teams/eyJkb21haW4iOiAid2VrYS5pbyIsICJzaWduYXR1cmVfaWQiOiA0ODY3NTgyMjg3OTM3NTM2LCAidXNlcl9lbWFpbCI6ICJiYXJ1Y2hAd2VrYS5pbyIsICJwbGFuIjogIlBsYXRpbnVtIn0=.png" alt=""></td></tr></tbody></table></div></div>