[dpdk-dev] [PATCH] vhost: add support to large linear mbufs

[Shahaf Shuler]

Wednesday, October 2, 2019 3:59 PM, Flavio Leitner:
> Obrembski MichalX <michalx.obrembski at intel.com>; Stokes Ian
> <ian.stokes at intel.com>
> Subject: Re: [dpdk-dev] [PATCH] vhost: add support to large linear mbufs
> 
> 
> Hi Shahaf,
> 
> Thanks for looking into this, see my inline comments.
> 
> On Wed, 2 Oct 2019 09:00:11 +0000
> Shahaf Shuler <shahafs at mellanox.com> wrote:
> 
> > Wednesday, October 2, 2019 11:05 AM, David Marchand:
> > > Subject: Re: [dpdk-dev] [PATCH] vhost: add support to large linear
> > > mbufs
> > >
> > > Hello Shahaf,
> > >
> > > On Wed, Oct 2, 2019 at 6:46 AM Shahaf Shuler <shahafs at mellanox.com>
> > > wrote:
> > > >

[...]

> > >
> > > I am missing some piece here.
> > > Which pool would the PMD take those external buffers from?
> >
> > The mbuf is always taken from the single mempool associated w/ the
> > rxq. The buffer for the mbuf may be allocated (in case virtio payload
> > is bigger than current mbuf size) from DPDK hugepages or any other
> > system memory and be attached to the mbuf.
> >
> > You can see example implementation of it in mlx5 PMD (checkout
> > rte_pktmbuf_attach_extbuf call)
> 
> Thanks, I wasn't aware of external buffers.
> 
> I see that attaching external buffers of the correct size would be more
> efficient in terms of saving memory/avoiding sparsing.
> 
> However, we still need to be prepared to the worse case scenario (all
> packets 64K), so that doesn't help with the total memory required.

Am not sure why. 
The allocation can be per demand. That is - only when you encounter a large buffer. 

Having buffer allocated in advance will benefit only from removing the cost of the rte_*malloc. However on such big buffers, and further more w/ device offloads like TSO, am not sure that is an issue. 

> 
> The current patch pushes the decision to the application which knows better
> the workload. If more memory is available, it can optionally use large buffers,
> otherwise just don't pass that. Or even decide whether to share the same
> 64K mempool between multiple vhost ports or use one mempool per port.
> 
> Perhaps I missed something, but managing memory with mempool still
> require us to have buffers of 64K regardless if the data consumes less space.
> Otherwise the application or the PMD will have to manage memory itself.
> 
> If we let the PMD manages the memory, what happens if a port/queue is
> closed and one or more buffers are still in use (switching)? I don't see how to
> solve this cleanly.

Closing of the dev should return EBUSY till all buffers are free. 
What is the use case of closing a port while still having packet pending on other port of the switch? And why we cannot wait for them to complete transmission? 

> 
> fbl
> 
> >
> > >
> > > If it is from an additional mempool passed to the vhost pmd, I can't
> > > see the difference with Flavio proposal.
> > >
> > >
> > > > The pros of this approach is that you have full flexibility on the
> > > > memory
> > > allocation, and therefore a lower footprint.
> > > > The cons is the OVS will need to know how to handle mbuf w/
> > > > external
> > > buffers (not too complex IMO).
> > >
> > >
> > > --
> > > David Marchand