[dpdk-dev] [RFC] ethdev: introduce DMA memory mapping for external memory

[Shahaf Shuler]

Monday, November 19, 2018 7:18 PM, Burakov, Anatoly:
> Subject: Re: [RFC] ethdev: introduce DMA memory mapping for external
> memory
> 
> On 19-Nov-18 11:20 AM, Shahaf Shuler wrote:
> > Thursday, November 15, 2018 1:00 PM, Burakov, Anatoly:
> >> Subject: Re: [RFC] ethdev: introduce DMA memory mapping for external
> >> memory
> >>
> >> On 15-Nov-18 9:46 AM, Shahaf Shuler wrote:
> >>> Wednesday, November 14, 2018 7:06 PM, Burakov, Anatoly:
> >>>> Subject: Re: [RFC] ethdev: introduce DMA memory mapping for
> >>>> external memory
> >>>>
> >>>> On 14-Nov-18 2:53 PM, Shahaf Shuler wrote:
> >>>>> Hi Anatoly,
> >>>>>
> >>>>> Wednesday, November 14, 2018 1:19 PM, Burakov, Anatoly:
> >>>>>> Subject: Re: [RFC] ethdev: introduce DMA memory mapping for
> >>>>>> external memory
> >>>>>>

[...]

> >>
> >> I have to be honest, I didn't consider this question before :D I
> >> guess there could be cases where using rte_malloc might not be
> >> suitable because it wastes some memory on malloc elements, i.e. if
> >> you want to use N pages as memory, you'd have to allocate N+1 pages.
> >> If memory is at a premium, maybe manual management of it would be
> better in some cases.
> >
> > I had similar thoughts, more related to the usability from the user side.
> > When application allocated allocates external memory it just wants to use it
> for DMA, i.e. put it as the mbuf buf_addr or to populate it w/ a mempool.
> > It is an "overhead" to create a socket for this external memory, to populate
> it w/ the memory, and later on to malloc from this socket (or use the socket
> id for the mempool creation).
> > Not to mention the fact that maybe the application wants to manage this
> memory differently than how rte_malloc does.
> >
> > On the other hand, mapping memory to device before using it for dma is
> far more intuitive.
> 
> It is far more intuitive *if* you're doing all of the memory management
> yourself or "just" using this memory for a mempool. This was already working
> before, and if you had that as your use case, there is no need for the
> external memory feature.

I think it was broken in some way when you did the first work on the new memory management. 
Because after this work, PMDs were adjusted to look for the memory only in the fbarrays, and such external memory doesn't exists there.
otherwise, if working, why you added the support for the external memory? Which other use case you tried to cover? 

It is fixed in some way with the new work to support external memory, but requires changes in the application (the "overhead" I referred before). 

> 
> On the other hand, if you were to use it in a different way - for example,
> allocating hash tables or other DPDK data structures - then such a feature is
> essential. 

Hash tables that the NIC needs to read/write to/from?
I didn't get the point here. 

The entire point was to allow using external memory with
> semantics identical to how you use the rest of DPDK.

I understand this design choice, and it has the benefit of consistency v.s. more API call on the application side. 

> 
> Also, whether it's "intuitive" depends on perspective - you say "i expect to
> allocate memory and map it for DMA myself", i say "why do i have to care
> about DMA mapping, DPDK should do this for me" :) 

Right 😊. We should be driven by customers and not vendors, too bad there is no much feedback from them on the community. 
I know from VPP side they prefer not to use the rte_socket rather to manage the memory by them self, and they are perfectly fine with mapping it explicitly. This is true for other company that has vswitch solution (which I cannot mention by name). 


If you are using your
> own memory management and doing everything manually - you get to map
> everything for DMA manually. If you are using DPDK facilities
> - it is intuitive that for any DPDK-managed memory, internal or external,
> same rules apply across the board - DMA mapping included.

So if understand you point here, your claim "we need both". One for users which prefer the DPDK to do that for them and the other for the users who wants more control.
However, when keeping the API of explicit mapping, we can no longer say that external memory is behaves exactly the same as internal.

So going back to my RFC 😊, if we decide to keep both approach, what I suggest is to change the VFIO mapping one to be more generic and per device (can be rte_device).
I think the biggest questions we need to answer are
1. are we OK with generic, vendor agnostic API?
2.  are we OK that the explicit mapping is done per device?


> 
> >
> >>
> >>>
> >>>>
> >>>>>
> >>>>> I guess we can we can add a flag on the device mapping which will
> >>>>> say
> >>>> MAP_TO_ALL_DEVICES, to ease the application life in the presence of
> >>>> multiple device in the host.
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Thanks,
> >>>> Anatoly
> >>
> >>
> >> --
> >> Thanks,
> >> Anatoly
> 
> 
> --
> Thanks,
> Anatoly