[dpdk-dev] [RFC v3 0/1] Compression API in DPDK

Ahmed Mansour ahmed.mansour at nxp.com
Mon Dec 18 22:07:03 CET 2017


Hi Fiona,

On 12/15/2017 11:16 PM, Trahe, Fiona wrote:

> With the vast amounts of data being transported around networks and stored in
> storage systems, reducing data size is becoming ever more important. There
> are both software libraries and hardware devices available that provide
> compression, but no common API. This RFC proposes a compression API for
> DPDK to address this need.
>
> Features:
> • Deflate Algorithm (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc1951&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YCL49FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0)
> • LZS algorithm (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc2395&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Spk8bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0)
> • Static and Dynamic Huffman encoding.
> • Compression levels
> • Checksum generation
> • Asynchronous burst API
> • Session-based (a session contains immutable data only and is useable across devices)
> • stream-based to maintain state and history data for stateful flows.
>
> Note 1: Split of functionality above/below API
> When considering whether features should be supported on the API or not, the
> decision was based on the following:
> The purpose of the API is to decouple the application from the compute-intensive
> functions needed for compression by abstracting them under a common API. These
> can then be implemented by either hardware accelerators or optimised software
> libraries. Where features are not compute-intensive and unlikely to be
> offloaded or optimised, there’s nothing to be gained by each PMD having
> to separately implement them, and it makes more sense for them to be done
> above the API. So the following are not handled on the API and can be done above.
> • Prepending/appending protocol headers (gzip, zlib)

Agreed with the notion, however the header and footer handling can be 
added as an option. PMDs can support or not support each format. We 
(NXP) support auto padding of gzip and zlib headers as well as DEFLATE only.

> • File-handling, breaking files up into packets, reassembling.
> • Synchronous API
> • Serialisation of stateful requests
>
>
Is stateful planned for next phase? During design discussions we uncovered many API design
considerations necessary for stateful use. Chained stateful support in the future might not be
possible without compatibility breaking  

> Note 2: The tricky question of where the API belongs
> We considered
> 1. Extending cryptodev
> 2. New acceldev APIs for device handling + compressdev APIs for data path
> 3. New acceldev for all APIs
> 4. New compressdev API
> We've gone with option 4, a compressdev API.  See original RFC [1] for reasons.
> We explored wrapping this around a generic acceldev that would be hidden from the API
> but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
> but this added complexity and indirection and didn't add enough value, so we've abandoned it.
>
Makes sense. compression is common enough to be attempted to be a 
different device category.


> Opens:
>  - Define structures and API for proposed hash functionality
>  - Agree on stateful behaviour

What are the the current thoughts for stateful behavior?

>  - Complete capability APIs

A capability API is very much required as different HW/SW can have 
different capabilities.


regards,
Ahmed




More information about the dev mailing list