[dpdk-dev] [RFC v3 0/1] Compression API in DPDK

[Trahe, Fiona]

Hi Ahmed, 
thanks for your feedback and sorry for the slow response.
Comments below.

> -----Original Message-----
> From: Ahmed Mansour [mailto:ahmed.mansour at nxp.com]
> Sent: Monday, December 18, 2017 9:07 PM
> To: dev at dpdk.org; Shally.Verma at cavium.com; Hemant Agrawal <hemant.agrawal at nxp.com>
> Cc: Hemant Agrawal <hemant.agrawal at nxp.com>; Mahipal.Challa at cavium.com; Trahe, Fiona
> <fiona.trahe at intel.com>; NarayanaPrasad.Athreya at cavium.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch at intel.com>; Roy Pledge <roy.pledge at nxp.com>; Youri Querry
> <youri.querry_1 at nxp.com>
> Subject: Re: [RFC v3 0/1] Compression API in DPDK
> 
> Hi Fiona,
> 
> On 12/15/2017 11:16 PM, Trahe, Fiona wrote:
> 
> > With the vast amounts of data being transported around networks and stored in
> > storage systems, reducing data size is becoming ever more important. There
> > are both software libraries and hardware devices available that provide
> > compression, but no common API. This RFC proposes a compression API for
> > DPDK to address this need.
> >
> > Features:
> > • Deflate Algorithm
> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc195
> 1&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea
> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YCL4
> 9FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0)
> > • LZS algorithm
> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc239
> 5&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea
> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Spk8
> bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0)
> > • Static and Dynamic Huffman encoding.
> > • Compression levels
> > • Checksum generation
> > • Asynchronous burst API
> > • Session-based (a session contains immutable data only and is useable across devices)
> > • stream-based to maintain state and history data for stateful flows.
> >
> > Note 1: Split of functionality above/below API
> > When considering whether features should be supported on the API or not, the
> > decision was based on the following:
> > The purpose of the API is to decouple the application from the compute-intensive
> > functions needed for compression by abstracting them under a common API. These
> > can then be implemented by either hardware accelerators or optimised software
> > libraries. Where features are not compute-intensive and unlikely to be
> > offloaded or optimised, there’s nothing to be gained by each PMD having
> > to separately implement them, and it makes more sense for them to be done
> > above the API. So the following are not handled on the API and can be done above.
> > • Prepending/appending protocol headers (gzip, zlib)
> 
> Agreed with the notion, however the header and footer handling can be
> added as an option. PMDs can support or not support each format. We
> (NXP) support auto padding of gzip and zlib headers as well as DEFLATE only.
[Fiona] We'd like to stabilise the API with the current functionality before adding new features. Our focus now is on delivering a v1 of the full code rather than another iteration of the RFC.
The API will be experimental initially, so I think it should be easy add these later.
 
> > • File-handling, breaking files up into packets, reassembling.
> > • Synchronous API
> > • Serialisation of stateful requests
> >
> >
> Is stateful planned for next phase? During design discussions we uncovered many API design
> considerations necessary for stateful use. Chained stateful support in the future might not be
> possible without compatibility breaking
> 
[Fiona] Stateful is covered in the v3 version. See the thread 
http://dpdk.org/ml/archives/dev/2017-December/084713.html
Have a look at this, though maybe it would be better to hold off on a reply until the v2 version of this doc is posted by Shally.


> > Note 2: The tricky question of where the API belongs
> > We considered
> > 1. Extending cryptodev
> > 2. New acceldev APIs for device handling + compressdev APIs for data path
> > 3. New acceldev for all APIs
> > 4. New compressdev API
> > We've gone with option 4, a compressdev API.  See original RFC [1] for reasons.
> > We explored wrapping this around a generic acceldev that would be hidden from the API
> > but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
> > but this added complexity and indirection and didn't add enough value, so we've abandoned it.
> >
> Makes sense. compression is common enough to be attempted to be a
> different device category.
> 
> 
> > Opens:
> >  - Define structures and API for proposed hash functionality
> >  - Agree on stateful behaviour
> 
> What are the the current thoughts for stateful behavior?
> 
> >  - Complete capability APIs
> 
> A capability API is very much required as different HW/SW can have
> different capabilities.
[Fiona] Agreed. We have work to do to flesh this out  - but expect to do in v1 of actual code
rather than a further RFC iteration

> 
> regards,
> Ahmed
>