[dpdk-dev] dmadev discussion summary

[Jerin Jacob]

On Sat, Jul 3, 2021 at 5:30 PM Morten Brørup <mb at smartsharesystems.com> wrote:
>
> > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of fengchengwen
> > Sent: Saturday, 3 July 2021 11.45
> >
> > On 2021/7/3 16:53, Morten Brørup wrote:
> > >> From: fengchengwen [mailto:fengchengwen at huawei.com]
> > >> Sent: Saturday, 3 July 2021 02.32
> > >>
> > >> On 2021/7/2 22:57, Morten Brørup wrote:
> > >>>> In the DPDK framework, many data-plane API names contain queues.
> > >> e.g.
> > >>>> eventdev/crypto..
> > >>>> The concept of virt queues has continuity.
> > >>>
> > >>> I was also wondering about the name "virtual queue".
> > >>>
> > >>> Usually, something "virtual" would be an abstraction of something
> > >> physical, e.g. a software layer on top of something physical.
> > >>>
> > >>> Back in the days, a "DMA channel" used to mean a DMA engine on a
> > CPU.
> > >> If a CPU had 2 DMA channels, they could both be set up
> > simultaneously.
> > >>>
> > >>> The current design has the "dmadev" representing a CPU or other
> > chip,
> > >> which has one or more "HW-queues" representing DMA channels (of the
> > >> same type), and then "virt-queue" as a software abstraction on top,
> > for
> > >> using a DMA channel in different ways through individually
> > configured
> > >> contexts (virt-queues).
> > >>>
> > >>> It makes sense to me, although I would consider renaming "HW-queue"
> > >> to "channel" and perhaps "virt-queue" to "queue".
> > >>
> > >> The 'DMA channel' is more used than 'DMA queue', at least google
> > show
> > >> that there are at least 20+ times more.
> > >>
> > >> It's a good idea build the abstraction layer: queue <> channel <>
> > dma-
> > >> controller.
> > >> In this way, the meaning of each layer is relatively easy to
> > >> distinguish literally.
> > >>
> > >> will fix in V2
> > >>
> > >
> > > After re-reading all the mails in this thread, I have found one more
> > important high level detail still not decided:
> > >
> > > Bruce had suggested flattening the DMA channels, so each dmadev
> > represents a DMA channel. And DMA controllers with multiple DMA
> > channels will have to instantiate multiple dmadevs, one for each DMA
> > channel.
> >
> > The dpaa2_qdma support multiple DMA channels, I looked into the
> > dpaa2_qdma
> > and found the control-plane interacts with the kernel, so if we use the
> > flattening model, there maybe interactions between dmadevs.
>
> It is perfectly acceptable for the control-plane DMA controller functions to interact across multiple dmadevs, not being thread safe and using locks etc. to protect critical regions accessing shared registers.
>
> The key question is: Can the data-plane dmadev functions (rte_dma_copy() etc.) be implemented to be thread safe, so multiple threads can use data-plane dmadev functions simultaneously?

It can . if we need it in that way.

>
> >
> > >
> > > Just like a four port NIC instantiates four ethdevs.
> > >
> > > Then, like ethdevs, there would only be two abstraction layers:
> > dmadev <> queue, where a dmadev is a DMA channel on a DMA controller.
> >
> > the dmadev <> channel <> queue model, there are three abstraction
> > layers,
> > and two abstraction layers.
> >
> > >
> > > However, this assumes that the fast path functions on the individual
> > DMA channels of a DMA controller can be accessed completely
> > independently and simultaneously by multiple threads. (Otherwise, the
> > driver would need to implement critical regions or locking around
> > accessing the common registers in the DMA controller shared by the DMA
> > channels.)
> >
> > Yes, this scheme has a big implicit dependency, that is, the channels
> > are
> > independent of each other.
> >
> > >
> > > Unless any of the DMA controller vendors claim that this assumption
> > about independence of the DMA channels is wrong, I strongly support
> > Bruce's flattening suggestion.
> > >
> > > -Morten
> > >
> >
>