<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-GB" link="#0563C1" vlink="#954F72"> <div class="WordSection1"> What happens when the Nvidia GPU driver kernel callback occurs to invalidate the pinned GPU memory region? Doesn’t the NIC need to cease all DMA transfers to/from that region before the kernel callback can complete?<o:p></o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Elena Agostini <eagostini@nvidia.com> Sent: 04 January 2022 13:55 To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net> Cc: dev@dpdk.org Subject: Re: [PATCH v2] gpudev: pin GPU memory<o:p></o:p> </div> </div> <o:p> </o:p> <div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt"> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p> </div> <div> <div> > 04/01/2022 03:41, <a href="mailto:eagostini@nvidia.com"> eagostini@nvidia.com</a>:<o:p></o:p> > > From: Elena Agostini <<a href="mailto:eagostini@nvidia.com">eagostini@nvidia.com</a>><o:p></o:p> > ><o:p></o:p> > > Enable the possibility to make a GPU memory area accessible from<o:p></o:p> > > the CPU.<o:p></o:p> > ><o:p></o:p> > > GPU memory has to be allocated via rte_gpu_mem_alloc().<o:p></o:p> > ><o:p></o:p> > > This patch allows the gpudev library to pin, through the GPU driver,<o:p></o:p> > > a chunk of GPU memory and to return a memory pointer usable<o:p></o:p> > > by the CPU to access the GPU memory area.<o:p></o:p> > ><o:p></o:p> > > Signed-off-by: Elena Agostini <<a href="mailto:eagostini@nvidia.com">eagostini@nvidia.com</a>><o:p></o:p> > [...]<o:p></o:p> > > +/**<o:p></o:p> > > + * @warning<o:p></o:p> > > + * @b EXPERIMENTAL: this API may change without prior notice.<o:p></o:p> > > + *<o:p></o:p> > > + * Pin a chunk of GPU memory to make it accessible from the CPU<o:p></o:p> > <o:p></o:p> > You should define what means "pin" exactly.<o:p></o:p> > Which properties should we expect?<o:p></o:p> > <o:p></o:p> <o:p> </o:p> Thanks for reviewing, this is the kind of discussion I wanted to have.<o:p></o:p> Maybe "pin" is too GDRCopy specific oriented.<o:p></o:p> Here I want to make a GPU memory buffer visible from the CPU. In case<o:p></o:p> of NVIDIA, this means the GPU memory address has to be pinned (virtual address<o:p></o:p> doesn't change) and dma-mapped.<o:p></o:p> <o:p> </o:p> Maybe the name should be more like rte_gpu_mem_to_cpu() that's more<o:p></o:p> explicative and generic.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > > + * using the memory pointer returned by the function.<o:p></o:p> > <o:p></o:p> > Which function should return the pointer?<o:p></o:p> > rte_gpu_mem_pin is returning an int.<o:p></o:p> <o:p> </o:p> Oversight, will fix it.<o:p></o:p> <o:p> </o:p> > <o:p></o:p> > <o:p></o:p> > > + * GPU memory has to be allocated via rte_gpu_mem_alloc().<o:p></o:p> > <o:p></o:p> > Why pinning is not done by rte_gpu_mem_alloc()?<o:p></o:p> > Should it be a flag?<o:p></o:p> <o:p> </o:p> rte_gpu_mem_alloc() allocate virtual memory on the GPU that doesn't have<o:p></o:p> to be necessarily shared (pinned) to make it visible from CPU.<o:p></o:p> <o:p> </o:p> > <o:p></o:p> > > + *<o:p></o:p> > > + * @param dev_id<o:p></o:p> > > + * Device ID requiring pinned memory.<o:p></o:p> > > + * @param size<o:p></o:p> > > + * Number of bytes to pin.<o:p></o:p> > > + * Requesting 0 will do nothing.<o:p></o:p> > > + * @param ptr<o:p></o:p> > > + * Pointer to the GPU memory area to be pinned.<o:p></o:p> > > + * NULL is a no-op accepted value.<o:p></o:p> > > +<o:p></o:p> > > + * @return<o:p></o:p> > > + * A pointer to the pinned GPU memory usable by the CPU, otherwise NULL and rte_errno is set:<o:p></o:p> > > + * - ENODEV if invalid dev_id<o:p></o:p> > > + * - EINVAL if reserved flags<o:p></o:p> > <o:p></o:p> > Which reserved flags?<o:p></o:p> > <o:p></o:p> > > + * - ENOTSUP if operation not supported by the driver<o:p></o:p> > > + * - E2BIG if size is higher than limit<o:p></o:p> > > + * - ENOMEM if out of space<o:p></o:p> > <o:p></o:p> > Is out of space relevant for pinning?<o:p></o:p> <o:p> </o:p> Yes, let me add it<o:p></o:p> <o:p> </o:p> > <o:p></o:p> > > + * - EPERM if driver error<o:p></o:p> > > + */<o:p></o:p> > > +__rte_experimental<o:p></o:p> > > +int rte_gpu_mem_pin(int16_t dev_id, size_t size, void *ptr);<o:p></o:p> </div> </div> </div> </div> </body> </html>