[dpdk-dev] [PATCH] compressdev: implement API - mbuf alternative

[Verma, Shally]

HI Fiona

So I understand we're moving away from mbufs because of its size limitation (64k) and cacheline overhead and their more suitability to n/w applications. Given that, I understand benefit of having another structure to input data but then what is proposal for ipcomp like application where mbuf usage may be a better option? Should we keep support for both (mbuf and this structure) so that apps can use appropriate data structure depending on their requirement.

Further comments, on github.

Thanks
Shally

>-----Original Message-----
>From: Trahe, Fiona [mailto:fiona.trahe at intel.com]
>Sent: 12 March 2018 21:31
>To: Ahmed Mansour <ahmed.mansour at nxp.com>; Verma, Shally <Shally.Verma at cavium.com>; dev at dpdk.org
>Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch at intel.com>; Athreya, Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>;
>Gupta, Ashish <Ashish.Gupta at cavium.com>; Sahu, Sunila <Sunila.Sahu at cavium.com>; Challa, Mahipal
><Mahipal.Challa at cavium.com>; Jain, Deepak K <deepak.k.jain at intel.com>; Hemant Agrawal <hemant.agrawal at nxp.com>; Roy
>Pledge <roy.pledge at nxp.com>; Youri Querry <youri.querry_1 at nxp.com>; fiona.trahe at gmail.com; Daly, Lee <lee.daly at intel.com>;
>Jozwiak, TomaszX <tomaszx.jozwiak at intel.com>
>Subject: RE: [dpdk-dev] [PATCH] compressdev: implement API - mbuf alternative
>
>Hi Shally, Ahmed, and anyone else interested in compressdev,
>
>I mentioned last week that we've been exploring using something other than mbufs to pass src/dst buffers to compressdev PMDs.
>
>Reasons:
> - mbuf data is limited to 64k-1 in each segment of a chained mbuf. Data for compression
>    can be greater and it would add cycles to have to break up into smaller segments.
> - data may originate in mbufs, but is more likely, particularly for storage use-cases,  to
>    originate in other data structures.
> - There's a 2 cache-line overhead for every segment in a chain, most of this data
>    is network-related, not needed by compressdev
>So moving to a custom structure would minimise memory overhead, remove restriction on 64k-1 size and give more flexibility if
>compressdev ever needs any comp-specific meta-data.
>
>We've come up with a compressdev-specific structure using the struct iovec from sys/uio.h, which is commonly used by storage
>applications. This would replace the src and dest mbufs in the  op.
>I'll not include the code here - Pablo will push that to github shortly and we'd appreciate review comments there.
>https://github.com/pablodelara/dpdk-draft-compressdev
>Just posting on the mailing list to give a heads-up and ensure this reaches a wider audience than may see it on github.
>
>Note : We also considered having no data structures in the op, instead the application
>would supply a callback which the PMD would use to retrieve meta-data (virt address, iova, length)
>for each next segment as needed. While this is quite flexible and allow the application
>to keep its data in its native structures, it's likely to cost more cycles.
>So we're not proposing this at the moment, but hope to benchmark it later while the API is still experimental.
>
>General feedback on direction is welcome here on the mailing list.
>For feedback on the details of implementation we would appreciate comments on github.
>
>Regards,
>Fiona.