[EXTERNAL] Re: [RFC] lib/dma: introduce inter-process and inter-OS DMA

[Vamsi Krishna Attunuru]

Hi Feng, Anatoly

Could please review below proposal and share your thoughts.

Regards
Vamsi

>> -----Original Message-----
>> From: Vamsi Krishna Attunuru <vattunuru at marvell.com>
>> >From: Vamsi Krishna Attunuru
>> >Sent: Monday, September 22, 2025 5:19 PM
>> >To: fengchengwen <fengchengwen at huawei.com>; dev at dpdk.org;
>> >anatoly.burakov at intel.com
>> >Cc: thomas at monjalon.net; bruce.richardson at intel.com;
>> >vladimir.medvedkin at intel.com; anatoly.burakov at intel.com;
>> >kevin.laatz at intel.com; Jerin Jacob <jerinj at marvell.com>
>> >Subject: RE: [EXTERNAL] Re: [RFC] lib/dma: introduce inter-process
>> >and
>> >inter- OS DMA
>> >
>> >Hi Feng,
>> >
>> >>Hi Vamsi, This commit change is more than discussed, it add control
>> >>API which for group management. 1. Control API: I check this commit
>> >>and Intel commit [1], it seem has a quite difference. I hope Intel
>> >>guys can express views ZjQcmQRYFpfptBannerEnd Hi Vamsi,
>> >>
>> >>This commit change is more than discussed, it add control API which
>> >>for group management.
>> >>
>> >>1. Control API: I check this commit and Intel commit [1], it seem
>> >>has a quite difference.
>> >>   I hope Intel guys can express views. I prefer not add this part
>> >>if no
>> >response.
>> >
>> >This new feature needs to be securely managed through control APIs.
>> >It would be extremely helpful if the folks at Intel and you as well
>> >could provide support or inputs on this.
>
>
>Beyond adding Intel folks to this thread, I don't see any further steps we can
>take to drive review at this stage.
>
>That said, the table-based concept used in the current API may not be
>portable, and we may need improvements here.
>
>Based on my understanding, DMA devices used for inter process copy can be
>classified into three categories:
>
>Class A: Requires a pair of DMA devices (one on each end of process/domain)
>for data transfer. Marvell DMA devices fall into this category.
>Class B: Requires only a single DMA device (one process/domain has a DMA
>device, the other process does not). Intel DMA devices fall here.
>Class C: Other types of devices that we are not yet aware of.
>
>Abstracting all of these under a single API will be challenging. Linux and other
>OSes do not provide control-plane APIs for this, so DPDK must provide control
>plane mechanisms to support Class A, Class B, and Class C devices.
>
>Proposal: Split development into separate sets:
>-----------------------------------------------
>Set A: Focus only on the datapath. Assume uint16_t *src_handle and uint16_t
>*dst_handle come from elsewhere (Class C).
>Set B: Introduce capabilities for Class A devices with portable APIs (proposal
>below, without table concept).
>Set C: Introduce capabilities for Class B devices and relevant APIs, to be added
>when needed.
>
>We can merge Set A in the current release and move Set B to a next release
>_if_ review or support for Class A devices requires more time.
>
>@fengchengwen Thoughts?
>
>Class A API Proposal:
>---------------------
>These APIs are based on a new capability flag for inter-process or inter-OS
>DMA transfers for Class A devices.
>
>
>/** Creates an access group for pair-type inter-process or inter-OS DMA
>transfers. */ int rte_dma_access_pair_group_create(const struct
>rte_dma_dev *dev,
>                                     rte_uuid_t process_id,
>                                     rte_uuid_t token,
>                                     uint16_t *group_id);
>
>/** Destroys an access group once all participating devices have exited. */ int
>rte_dma_access_pair_group_destroy(const struct rte_dma_dev *dev,
>                                      uint16_t group_id);
>
>/** Allows a device to join an existing access group using a device handle and
>token. */ int rte_dma_access_pair_group_join(const struct rte_dma_dev
>*dev,
>                                   uint16_t group_id,
>                                   rte_uuid_t process_id,
>                                   rte_uuid_t token,
>                                   rte_dma_access_pair_leave_cb_t leave_cb);
>
>/** Removes a device from an access group. */ int
>rte_dma_access_pair_group_leave(const struct rte_dma_dev *dev,
>                                    uint16_t group_id);
>
>/** Retrieves the source and destination handles for a given device within the
>group. */ int rte_dma_access_pair_gorup_src_dst_handles_get(const struct
>rte_dma_dev *dev,
>                                            uint16_t group_id,
>                                            rte_uuid_t src_process_id,
>                                            rte_uuid_t dst_process_id,
>                                            uint16_t *src_handle,
>                                            uint16_t *dst_handle);
>
>
>Parameters that need explanation:
>--------------------------------
>process_id: Unique ID for the process, generated via rte_uuid_ APIs
>token: Provided by an administrative actor to grant access, similar to VFIO VF
>token creation used in VFIO PF driver.
>leave_cb: Callback to notify when another side leaves the group
>
>
>Example Workflow for Class A Inter-Domain DMA Transfer:
>-------------------------------------------------------
>
>This example demonstrates how three processes — p0, p1, and p2 —
>coordinate inter-domain DMA transfers using pair-type(Class A) DMA devices.
>
>Step 1: Group Creation (p0)
>Process p0 calls rte_dma_access_pair_group_create() with a unique process
>handle and token. A group_id is returned.
>
>Step 2: Group Sharing
>group_id and token are shared with p1 and p2 via IPC or shared memory.
>
>Step 3: Group Joining (p1 & p2)
>Processes p1 and p2 call rte_dma_access_pair_group_join() with their process
>id  and the shared token from admin
>
>Step 4: Handle Discovery
>Each process uses rte_dma_access_pair_gorup_src_dst_handles_get() to
>retrieve source and destination handles for other processes.
>
>Step 5: Transfer Coordination
>Using the handles, each process configures a virtual channel (vchan) and
>initiates DMA transfers.
>
>Step 6: Group Teardown
>When a process no longer needs to participate, it calls
>rte_dma_access_pair_group_leave(). Other processes are notified via the
>registered callback with rte_dma_access_pair_group_join().
>Once all devices have exited, p0 calls rte_dma_access_pair_group_destroy()
>to clean up.
>
>
>For Class B: We can add new capability flag and have new set of APIs
>rte_dma_access_master_ or so. When such devices comes or when intel
>wants to add it
>
>
>
>