[PATCH v2] gpudev: pin GPU memory

Elena Agostini eagostini at nvidia.com
Tue Jan 4 18:53:12 CET 2022
Previous message (by thread): [PATCH v2] gpudev: pin GPU memory
Next message (by thread): [PATCH v3] gpudev: expose GPU memory
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
This specific feature is not about NIC activity, it’s about make GPU memory visible and accessible to the CPU so there isn’t an asynchronous hidden invalidation procedure.
That’s the reason for my comment to change the name from rte_gpu_mem_pin() to rte_gpu_mem_to_cpu() (or similar).

The scenario you are proposing can happen but in a different context, this is not the case.

Thanks
EA

From: John Alexander <John.Alexander at datapath.co.uk>
Date: Tuesday, 4 January 2022 at 18:31
To: Elena Agostini <eagostini at nvidia.com>, NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas at monjalon.net>
Cc: dev at dpdk.org <dev at dpdk.org>
Subject: RE: [PATCH v2] gpudev: pin GPU memory
External email: Use caution opening links or attachments

What happens when the Nvidia GPU driver kernel callback occurs to invalidate the pinned GPU memory region?  Doesn’t the NIC need to cease all DMA transfers to/from that region before the kernel callback can complete?

John Alexander
Senior Software Engineer
[cid:image152483.jpg at 5CD90415.402D82AB]<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.datapath.co.uk%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9b3%2FEpEiezz%2B0nLDmm2pE5ZcLkEzNbW%2F%2BJ1o61q2J3k%3D&reserved=0>
Bemrose House, Bemrose Park, Wayzgoose Drive
,
Derby
,
DE21 6XQ
[cid:image839434.png at FD4260AC.D2B1B11C]
+44 (0)1332 294 441<tel:+44%20(0)1332%20294%20441>
 |
www.datapath.co.uk<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.datapath.co.uk%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9b3%2FEpEiezz%2B0nLDmm2pE5ZcLkEzNbW%2F%2BJ1o61q2J3k%3D&reserved=0>
[LinkedIn]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fdatapath-ltd&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0PTscJhiHEK27PKDjq2nNgH5oRe499xubLh4S705u4A%3D&reserved=0>
[Twitter]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.twitter.com%2Fdatapathltd&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CVia5Sq%2Fi2yOAGx5d8PSsF7wJsA2oVM%2BgySwg5E7p2U%3D&reserved=0>
[YouTube]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fdatapathderby&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Obz00eF3OgTdp3YI5ux2YFRdx6Q65keqlMXVY6QNXCM%3D&reserved=0>
[cid:image974625.png at E5A3F546.B8FD2EA9]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datapath.co.uk%2Fevents-and-webinars%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MO6M5IE0mAymLgzIZwGqPPxqftcmzH7Gy43ZOGYHwAs%3D&reserved=0>
[Vote for Datapath]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datapath.co.uk%2Fdatapath-shortlisted-for-coveted-industry-award%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=D0wKDJVR%2B9kSzyIvtxt5yOdKZPCb%2Fd48biairPlYqt4%3D&reserved=0>
Datapath Ltd.  Registered Number: 1609392.  Registered in England at Bemrose House, Bemrose Park, Wayzgoose Drive, Derby. DE21 6XQ.
From: Elena Agostini <eagostini at nvidia.com>
Sent: 04 January 2022 13:55
To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas at monjalon.net>
Cc: dev at dpdk.org
Subject: Re: [PATCH v2] gpudev: pin GPU memory

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 04/01/2022 03:41, eagostini at nvidia.com<mailto:eagostini at nvidia.com>:
> > From: Elena Agostini <eagostini at nvidia.com<mailto:eagostini at nvidia.com>>
> >
> > Enable the possibility to make a GPU memory area accessible from
> > the CPU.
> >
> > GPU memory has to be allocated via rte_gpu_mem_alloc().
> >
> > This patch allows the gpudev library to pin, through the GPU driver,
> > a chunk of GPU memory and to return a memory pointer usable
> > by the CPU to access the GPU memory area.
> >
> > Signed-off-by: Elena Agostini <eagostini at nvidia.com<mailto:eagostini at nvidia.com>>
> [...]
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Pin a chunk of GPU memory to make it accessible from the CPU
>
> You should define what means "pin" exactly.
> Which properties should we expect?
>

Thanks for reviewing, this is the kind of discussion I wanted to have.
Maybe "pin" is too GDRCopy specific oriented.
Here I want to make a GPU memory buffer visible from the CPU. In case
of NVIDIA, this means the GPU memory address has to be pinned (virtual address
doesn't change) and dma-mapped.

Maybe the name should be more like rte_gpu_mem_to_cpu() that's more
explicative and generic.


> > + * using the memory pointer returned by the function.
>
> Which function should return the pointer?
> rte_gpu_mem_pin is returning an int.

Oversight, will fix it.

>
>
> > + * GPU memory has to be allocated via rte_gpu_mem_alloc().
>
> Why pinning is not done by rte_gpu_mem_alloc()?
> Should it be a flag?

rte_gpu_mem_alloc() allocate virtual memory on the GPU that doesn't have
to be necessarily shared (pinned) to make it visible from CPU.

>
> > + *
> > + * @param dev_id
> > + *   Device ID requiring pinned memory.
> > + * @param size
> > + *   Number of bytes to pin.
> > + *   Requesting 0 will do nothing.
> > + * @param ptr
> > + *   Pointer to the GPU memory area to be pinned.
> > + *   NULL is a no-op accepted value.
> > +
> > + * @return
> > + *   A pointer to the pinned GPU memory usable by the CPU, otherwise NULL and rte_errno is set:
> > + *   - ENODEV if invalid dev_id
> > + *   - EINVAL if reserved flags
>
> Which reserved flags?
>
> > + *   - ENOTSUP if operation not supported by the driver
> > + *   - E2BIG if size is higher than limit
> > + *   - ENOMEM if out of space
>
> Is out of space relevant for pinning?

Yes, let me add it

>
> > + *   - EPERM if driver error
> > + */
> > +__rte_experimental
> > +int rte_gpu_mem_pin(int16_t dev_id, size_t size, void *ptr);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mails.dpdk.org/archives/dev/attachments/20220104/5a711242/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image152483.jpg
Type: image/jpeg
Size: 5403 bytes
Desc: image152483.jpg
URL: <http://mails.dpdk.org/archives/dev/attachments/20220104/5a711242/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image158572.png
Type: image/png
Size: 651 bytes
Desc: image158572.png
URL: <http://mails.dpdk.org/archives/dev/attachments/20220104/5a711242/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image416332.png
Type: image/png
Size: 620 bytes
Desc: image416332.png
URL: <http://mails.dpdk.org/archives/dev/attachments/20220104/5a711242/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image447896.png
Type: image/png
Size: 684 bytes
Desc: image447896.png
URL: <http://mails.dpdk.org/archives/dev/attachments/20220104/5a711242/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image509100.png
Type: image/png
Size: 24194 bytes
Desc: image509100.png
URL: <http://mails.dpdk.org/archives/dev/attachments/20220104/5a711242/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image839434.png
Type: image/png
Size: 335 bytes
Desc: image839434.png
URL: <http://mails.dpdk.org/archives/dev/attachments/20220104/5a711242/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image974625.png
Type: image/png
Size: 20712 bytes
Desc: image974625.png
URL: <http://mails.dpdk.org/archives/dev/attachments/20220104/5a711242/attachment-0011.png>
Previous message (by thread): [PATCH v2] gpudev: pin GPU memory
Next message (by thread): [PATCH v3] gpudev: expose GPU memory
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the dev mailing list