[dpdk-dev] [RFC v3 0/1] Compression API in DPDK

[Ahmed Mansour]

Hi Fiona,

Thanks for your response. I added comments inline

On 12/22/2017 8:44 AM, Trahe, Fiona wrote:
> Hi Ahmed, 
> thanks for your feedback and sorry for the slow response.
> Comments below.
>
>> -----Original Message-----
>> From: Ahmed Mansour [mailto:ahmed.mansour at nxp.com]
>> Sent: Monday, December 18, 2017 9:07 PM
>> To: dev at dpdk.org; Shally.Verma at cavium.com; Hemant Agrawal <hemant.agrawal at nxp.com>
>> Cc: Hemant Agrawal <hemant.agrawal at nxp.com>; Mahipal.Challa at cavium.com; Trahe, Fiona
>> <fiona.trahe at intel.com>; NarayanaPrasad.Athreya at cavium.com; De Lara Guarch, Pablo
>> <pablo.de.lara.guarch at intel.com>; Roy Pledge <roy.pledge at nxp.com>; Youri Querry
>> <youri.querry_1 at nxp.com>
>> Subject: Re: [RFC v3 0/1] Compression API in DPDK
>>
>> Hi Fiona,
>>
>> On 12/15/2017 11:16 PM, Trahe, Fiona wrote:
>>
>>> With the vast amounts of data being transported around networks and stored in
>>> storage systems, reducing data size is becoming ever more important. There
>>> are both software libraries and hardware devices available that provide
>>> compression, but no common API. This RFC proposes a compression API for
>>> DPDK to address this need.
>>>
>>> Features:
>>> • Deflate Algorithm
>> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc195
>> 1&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea
>> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YCL4
>> 9FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0)
>>> • LZS algorithm
>> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc239
>> 5&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea
>> 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Spk8
>> bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0)
>>> • Static and Dynamic Huffman encoding.
>>> • Compression levels
>>> • Checksum generation
>>> • Asynchronous burst API
>>> • Session-based (a session contains immutable data only and is useable across devices)
>>> • stream-based to maintain state and history data for stateful flows.
>>>
>>> Note 1: Split of functionality above/below API
>>> When considering whether features should be supported on the API or not, the
>>> decision was based on the following:
>>> The purpose of the API is to decouple the application from the compute-intensive
>>> functions needed for compression by abstracting them under a common API. These
>>> can then be implemented by either hardware accelerators or optimised software
>>> libraries. Where features are not compute-intensive and unlikely to be
>>> offloaded or optimised, there’s nothing to be gained by each PMD having
>>> to separately implement them, and it makes more sense for them to be done
>>> above the API. So the following are not handled on the API and can be done above.
>>> • Prepending/appending protocol headers (gzip, zlib)
>> Agreed with the notion, however the header and footer handling can be
>> added as an option. PMDs can support or not support each format. We
>> (NXP) support auto padding of gzip and zlib headers as well as DEFLATE only.
> [Fiona] We'd like to stabilise the API with the current functionality before adding new features. Our focus now is on delivering a v1 of the full code rather than another iteration of the RFC.
> The API will be experimental initially, so I think it should be easy add these later.

[Ahmed] The two modes of using the API conflict. If in this API we
assume that the application will package the compressed data by
themselves then maybe we should not provide that service now or in the
future.
On the other hand, most users are familiar with the zlib.net interface
which does the packaging of the headers and footers for the user. I
think most applications will have to write zlib/gzip packagining
software to use this API.
Therefore, It makes sense for DPDK to own that portion of the packaging
I think. I am not sure how to provide that service and simultaneously
provide both a gzip and a zlib packaged output. I guess for the subset
of users who
need both gzip and zlib packaging for the same data, they will have to
do their own packaging. Most other customers will just use the API to
select what format they want right away

>  
>>> • File-handling, breaking files up into packets, reassembling.
>>> • Synchronous API
>>> • Serialisation of stateful requests
>>>
>>>
>> Is stateful planned for next phase? During design discussions we uncovered many API design
>> considerations necessary for stateful use. Chained stateful support in the future might not be
>> possible without compatibility breaking
>>
> [Fiona] Stateful is covered in the v3 version. See the thread 
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpdk.org%2Fml%2Farchives%2Fdev%2F2017-December%2F084713.html&data=02%7C01%7Cahmed.mansour%40nxp.com%7Cc79921b3d15d468da38e08d549421eb5%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636495470754303375&sdata=xlQdD1hk4N%2BENbB6nNCmBXdR6ycgwyKdrckPpYR2DjE%3D&reserved=0
> Have a look at this, though maybe it would be better to hold off on a reply until the v2 version of this doc is posted by Shally.

[Ahmed] The proposed implementation only allows one op to be in flight
at a time for a particular stream. We see use cases where multiple
stateful in flight pieces of info are sent sequentially

>
>>> Note 2: The tricky question of where the API belongs
>>> We considered
>>> 1. Extending cryptodev
>>> 2. New acceldev APIs for device handling + compressdev APIs for data path
>>> 3. New acceldev for all APIs
>>> 4. New compressdev API
>>> We've gone with option 4, a compressdev API.  See original RFC [1] for reasons.
>>> We explored wrapping this around a generic acceldev that would be hidden from the API
>>> but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
>>> but this added complexity and indirection and didn't add enough value, so we've abandoned it.
>>>
>> Makes sense. compression is common enough to be attempted to be a
>> different device category.
>>
>>
>>> Opens:
>>>  - Define structures and API for proposed hash functionality
>>>  - Agree on stateful behaviour
>> What are the the current thoughts for stateful behavior?
>>
>>>  - Complete capability APIs
>> A capability API is very much required as different HW/SW can have
>> different capabilities.
> [Fiona] Agreed. We have work to do to flesh this out  - but expect to do in v1 of actual code
> rather than a further RFC iteration

[Ahmed] How far apart are we going to allow the different DPDK
implementations to drift? I understood that DPDK is meant to improve
portability.The compatibility API moves the design complextiy one layer up.
I think this will make it harder for application designers which may impact adoption

>> regards,
>> Ahmed
>>
>