<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="en-IT" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> This specific feature is not about NIC activity, it’s about make GPU memory visible and accessible to the CPU so there isn’t an asynchronous hidden invalidation procedure.<o:p></o:p> That’s the reason for my comment to change the name from rte_gpu_mem_pin() to rte_gpu_mem_to_cpu() (or similar).<o:p></o:p> <o:p> </o:p> The scenario you are proposing can happen but in a different context, this is not the case.<o:p></o:p> <o:p> </o:p> <div> <div> <div> Thanks<o:p></o:p> </div> </div> </div> EA<o:p></o:p> <o:p> </o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: John Alexander <John.Alexander@datapath.co.uk> Date: Tuesday, 4 January 2022 at 18:31 To: Elena Agostini <eagostini@nvidia.com>, NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net> Cc: dev@dpdk.org <dev@dpdk.org> Subject: RE: [PATCH v2] gpudev: pin GPU memory<o:p></o:p> </div> <table class="MsoNormalTable" border="1" cellpadding="0" style="margin-left:36.0pt;background:#FFEB9C"> <tbody> <tr> <td style="padding:.75pt .75pt .75pt .75pt"> External email: Use caution opening links or attachments <o:p></o:p> </td> </tr> </tbody> </table> <o:p> </o:p> <div> What happens when the Nvidia GPU driver kernel callback occurs to invalidate the pinned GPU memory region? Doesn’t the NIC need to cease all DMA transfers to/from that region before the kernel callback can complete?<o:p></o:p> <o:p></o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="101%" style="width:101.02%;margin-left:36.0pt"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> John Alexander<o:p></o:p> </td> </tr> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> Senior Software Engineer<o:p></o:p> </td> </tr> </tbody> </table> </td> </tr> <tr> <td valign="top" style="padding:9.75pt 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:4.5pt 0cm 4.5pt 0cm"> <a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.datapath.co.uk%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9b3%2FEpEiezz%2B0nLDmm2pE5ZcLkEzNbW%2F%2BJ1o61q2J3k%3D&reserved=0" target="_blank"><img border="0" width="201" height="56" style="width:2.0937in;height:.5833in" id="_x0000_i1031" src="cid:image152483.jpg@5CD90415.402D82AB"></a><o:p></o:p> </td> </tr> </tbody> </table> </td> </tr> <tr> <td valign="top" style="padding:9.75pt 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> Bemrose House, Bemrose Park, Wayzgoose Drive<o:p></o:p> </td> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> , <o:p></o:p> </td> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> Derby<o:p></o:p> </td> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> , <o:p></o:p> </td> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> DE21 6XQ<o:p></o:p> </td> </tr> </tbody> </table> </td> </tr> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 1.5pt 0cm 0cm"> <img border="0" width="18" height="18" style="width:.1875in;height:.1875in" id="_x0000_i1030" src="cid:image839434.png@FD4260AC.D2B1B11C"><o:p></o:p> </td> </tr> </tbody> </table> </td> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <a href="tel:+44%20(0)1332%20294%20441" target="_blank">+44 (0)1332 294 441</a><o:p></o:p> </td> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> | <o:p></o:p> </td> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.datapath.co.uk%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9b3%2FEpEiezz%2B0nLDmm2pE5ZcLkEzNbW%2F%2BJ1o61q2J3k%3D&reserved=0" target="_blank">www.datapath.co.uk</a><o:p></o:p> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </td> </tr> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:7.5pt 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 2.25pt 2.25pt 0cm"> <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fdatapath-ltd&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0PTscJhiHEK27PKDjq2nNgH5oRe499xubLh4S705u4A%3D&reserved=0" target="_blank"><img border="0" width="20" height="20" style="width:.2083in;height:.2083in" id="_x0000_i1029" src="cid:image158572.png@7940B28E.9299C76A" alt="LinkedIn"></a><o:p></o:p> </td> </tr> </tbody> </table> </td> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 2.25pt 2.25pt 0cm"> <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.twitter.com%2Fdatapathltd&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CVia5Sq%2Fi2yOAGx5d8PSsF7wJsA2oVM%2BgySwg5E7p2U%3D&reserved=0" target="_blank"><img border="0" width="20" height="20" style="width:.2083in;height:.2083in" id="_x0000_i1028" src="cid:image447896.png@A0AF8451.CDC641E2" alt="Twitter"></a><o:p></o:p> </td> </tr> </tbody> </table> </td> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 2.25pt 2.25pt 0cm"> <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fdatapathderby&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Obz00eF3OgTdp3YI5ux2YFRdx6Q65keqlMXVY6QNXCM%3D&reserved=0" target="_blank"><img border="0" width="20" height="20" style="width:.2083in;height:.2083in" id="_x0000_i1027" src="cid:image416332.png@1828A521.2A92D787" alt="YouTube"></a><o:p></o:p> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </td> </tr> <tr> <td valign="top" style="padding:12.0pt 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 15.0pt 0cm"> <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datapath.co.uk%2Fevents-and-webinars%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MO6M5IE0mAymLgzIZwGqPPxqftcmzH7Gy43ZOGYHwAs%3D&reserved=0" target="_blank"><img border="0" width="601" height="75" style="width:6.2604in;height:.7812in" id="_x0000_i1026" src="cid:image974625.png@E5A3F546.B8FD2EA9"></a><o:p></o:p> </td> </tr> </tbody> </table> </td> </tr> <tr> <td valign="top" style="padding:0cm 0cm 0cm 0cm"> <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datapath.co.uk%2Fdatapath-shortlisted-for-coveted-industry-award%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=D0wKDJVR%2B9kSzyIvtxt5yOdKZPCb%2Fd48biairPlYqt4%3D&reserved=0" target="_blank"><img border="0" width="601" height="75" style="width:6.2604in;height:.7812in" id="_x0000_i1025" src="cid:image509100.png@4785B606.24F2E4FA" alt="Vote for Datapath"></a><o:p></o:p> </td> </tr> </tbody> </table> </td> </tr> <tr> <td valign="top" style="padding:9.75pt 0cm 0cm 0cm"> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"> <tbody> <tr> <td valign="top" style="padding:0cm 0cm 3.0pt 0cm"> Datapath Ltd. Registered Number: 1609392. Registered in England at Bemrose House, Bemrose Park, Wayzgoose Drive, Derby. DE21 6XQ.<o:p></o:p> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Elena Agostini <eagostini@nvidia.com> Sent: 04 January 2022 13:55 To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net> Cc: dev@dpdk.org Subject: Re: [PATCH v2] gpudev: pin GPU memory<o:p></o:p> </div> </div> <o:p></o:p> <div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt"> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p> </div> <div> <div> > 04/01/2022 03:41, <a href="mailto:eagostini@nvidia.com">eagostini@nvidia.com</a>:<o:p></o:p> > > From: Elena Agostini <<a href="mailto:eagostini@nvidia.com">eagostini@nvidia.com</a>><o:p></o:p> > ><o:p></o:p> > > Enable the possibility to make a GPU memory area accessible from<o:p></o:p> > > the CPU.<o:p></o:p> > ><o:p></o:p> > > GPU memory has to be allocated via rte_gpu_mem_alloc().<o:p></o:p> > ><o:p></o:p> > > This patch allows the gpudev library to pin, through the GPU driver,<o:p></o:p> > > a chunk of GPU memory and to return a memory pointer usable<o:p></o:p> > > by the CPU to access the GPU memory area.<o:p></o:p> > ><o:p></o:p> > > Signed-off-by: Elena Agostini <<a href="mailto:eagostini@nvidia.com">eagostini@nvidia.com</a>><o:p></o:p> > [...]<o:p></o:p> > > +/**<o:p></o:p> > > + * @warning<o:p></o:p> > > + * @b EXPERIMENTAL: this API may change without prior notice.<o:p></o:p> > > + *<o:p></o:p> > > + * Pin a chunk of GPU memory to make it accessible from the CPU<o:p></o:p> > <o:p></o:p> > You should define what means "pin" exactly.<o:p></o:p> > Which properties should we expect?<o:p></o:p> > <o:p></o:p> <o:p></o:p> Thanks for reviewing, this is the kind of discussion I wanted to have.<o:p></o:p> Maybe "pin" is too GDRCopy specific oriented.<o:p></o:p> Here I want to make a GPU memory buffer visible from the CPU. In case<o:p></o:p> of NVIDIA, this means the GPU memory address has to be pinned (virtual address<o:p></o:p> doesn't change) and dma-mapped.<o:p></o:p> <o:p></o:p> Maybe the name should be more like rte_gpu_mem_to_cpu() that's more<o:p></o:p> explicative and generic.<o:p></o:p> <o:p></o:p> <o:p></o:p> > > + * using the memory pointer returned by the function.<o:p></o:p> > <o:p></o:p> > Which function should return the pointer?<o:p></o:p> > rte_gpu_mem_pin is returning an int.<o:p></o:p> <o:p></o:p> Oversight, will fix it.<o:p></o:p> <o:p></o:p> > <o:p></o:p> > <o:p></o:p> > > + * GPU memory has to be allocated via rte_gpu_mem_alloc().<o:p></o:p> > <o:p></o:p> > Why pinning is not done by rte_gpu_mem_alloc()?<o:p></o:p> > Should it be a flag?<o:p></o:p> <o:p></o:p> rte_gpu_mem_alloc() allocate virtual memory on the GPU that doesn't have<o:p></o:p> to be necessarily shared (pinned) to make it visible from CPU.<o:p></o:p> <o:p></o:p> > <o:p></o:p> > > + *<o:p></o:p> > > + * @param dev_id<o:p></o:p> > > + * Device ID requiring pinned memory.<o:p></o:p> > > + * @param size<o:p></o:p> > > + * Number of bytes to pin.<o:p></o:p> > > + * Requesting 0 will do nothing.<o:p></o:p> > > + * @param ptr<o:p></o:p> > > + * Pointer to the GPU memory area to be pinned.<o:p></o:p> > > + * NULL is a no-op accepted value.<o:p></o:p> > > +<o:p></o:p> > > + * @return<o:p></o:p> > > + * A pointer to the pinned GPU memory usable by the CPU, otherwise NULL and rte_errno is set:<o:p></o:p> > > + * - ENODEV if invalid dev_id<o:p></o:p> > > + * - EINVAL if reserved flags<o:p></o:p> > <o:p></o:p> > Which reserved flags?<o:p></o:p> > <o:p></o:p> > > + * - ENOTSUP if operation not supported by the driver<o:p></o:p> > > + * - E2BIG if size is higher than limit<o:p></o:p> > > + * - ENOMEM if out of space<o:p></o:p> > <o:p></o:p> > Is out of space relevant for pinning?<o:p></o:p> <o:p></o:p> Yes, let me add it<o:p></o:p> <o:p></o:p> > <o:p></o:p> > > + * - EPERM if driver error<o:p></o:p> > > + */<o:p></o:p> > > +__rte_experimental<o:p></o:p> > > +int rte_gpu_mem_pin(int16_t dev_id, size_t size, void *ptr);<o:p></o:p> </div> </div> </div> </div> </div> </body> </html>