[dpdk-dev] [RFC v1] doc compression API for DPDK

Verma, Shally Shally.Verma at cavium.com
Thu Dec 7 06:42:54 CET 2017


HI Fiona

> -----Original Message-----
> From: Trahe, Fiona [mailto:fiona.trahe at intel.com]
> Sent: 02 December 2017 00:42
> To: Verma, Shally <Shally.Verma at cavium.com>; dev at dpdk.org
> Cc: Athreya, Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>;
> Challa, Mahipal <Mahipal.Challa at cavium.com>; De Lara Guarch, Pablo
> <pablo.de.lara.guarch at intel.com>; Trahe, Fiona <fiona.trahe at intel.com>
> Subject: RE: [RFC v1] doc compression API for DPDK
> 
> 
> 
> > -----Original Message-----
> > From: Verma, Shally [mailto:Shally.Verma at cavium.com]
> > Sent: Thursday, November 30, 2017 11:13 AM
> > To: Trahe, Fiona <fiona.trahe at intel.com>; dev at dpdk.org
> > Cc: Athreya, Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>;
> Challa, Mahipal
> > <Mahipal.Challa at cavium.com>
> > Subject: Re: [RFC v1] doc compression API for DPDK
> >
> > HI Fiona
> >
> > > -----Original Message-----
> > > From: Trahe, Fiona [mailto:fiona.trahe at intel.com]
> > > Sent: 28 November 2017 00:25
> > > To: Verma, Shally <Shally.Verma at cavium.com>; dev at dpdk.org; Athreya,
> > > Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>; Challa,
> Mahipal
> > > <Mahipal.Challa at cavium.com>; De Lara Guarch, Pablo
> > > <pablo.de.lara.guarch at intel.com>
> > > Cc: Trahe, Fiona <fiona.trahe at intel.com>
> > > Subject: RE: [RFC v1] doc compression API for DPDK
> > >
> > > Hi Shally,
> > >
> > > > -----Original Message-----
> > > > From: Verma, Shally [mailto:Shally.Verma at cavium.com]
> > > > Sent: Tuesday, October 31, 2017 11:39 AM
> > > > To: dev at dpdk.org; Trahe, Fiona <fiona.trahe at intel.com>; Athreya,
> > > Narayana Prasad
> > > > <NarayanaPrasad.Athreya at cavium.com>; Challa, Mahipal
> > > <Mahipal.Challa at cavium.com>
> > > > Subject: [RFC v1] doc compression API for DPDK
> > > >
> > > > HI Fiona
> > > >
> > > > This is an RFC document to brief our understanding and requirements
> on
> > > compression API proposal in
> > > > DPDK. It is based on "[RFC] Compression API in DPDK
> > > http://dpdk.org/ml/archives/dev/2017-
> > > > October/079377.html".
> > > > Intention of this document is to align on concepts built into
> compression
> > > API, its usage and identify further
> > > > requirements.
> > > >
> > > > Going further it could be a base to Compression Module Programmer
> > > Guide.
> > > >
> > > > Current scope is limited to
> > > > - definition of the terminology which makes up foundation of
> compression
> > > API
> > > > - typical API flow expected to use by applications
> > > >
> > > > Overview
> > > > ~~~~~~~~
> > > > A. Notion of a session in compression API
> > > > ==================================
> > > > A Session is per device logical entity which is setup with chained-xforms
> to
> > > be performed on burst
> > > > operations where individual entry contains operation type
> > > (decompress/compress) and related parameter.
> > > > A typical Session parameter includes:
> > > > - compress / decompress
> > > > - dev_id
> > > > - compression algorithm and other related parameters
> > > > - mempool - for use by session for runtime requirement
> > > > - and any other associated private data maintained by session
> > > >
> > > > Application can setup multiple sessions on a device as dictated by
> > > dev_info.nb_sessions or
> > > > nb_session_per_qp.
> > > >
> > > [Fiona] The session design is modelled on the cryptodev session design
> and
> > > so allows
> > > to create a session which can be used on different driver types. E.g.  a
> > > session could be set up and initialised
> > > to run on a QuickAssist device and a Software device. This may be useful
> for
> > > stateless
> > > requests, and enable load-balancing. For stateful flows the session
> should be
> > > set up for
> > > only one specific driver-type as the state information will be stored in the
> > > private data specific to the driver-type
> > > and not transferrable between driver-types.
> > > So a session
> > >  - is not per-device
> > >  - has no dev_id
> > >  - has no mempool stored in it - the pool is created by the application, the
> lib
> > > can retrieve the pool from the object with rte_mempool_from_obj()
> > >  - does not have a limit number per device, just per qp, i.e. there is no
> > > dev_info.nb_sessions, just dev_info.max_nb_sessions_per_qp
> > >
> > > Do you think any of this needs to be changed?
> >
> > [Shally] Please help confirm following before I could answer this.
> >
> > In cryptodev, session holds an drivers array initialized on it where each can
> be setup to perform
> > same/different operation in its private_data.
> > On mapping it to compression it mean, a session:
> > - Will not retain any of the info as mentioned above (xform, mempool,
> algos et el). All such information is
> > maintained as part of associated device driver private data.
> [Fiona] exactly
> 
> > - App can use same session to set compress xform and decompress xform
> on devices but if both devices
> > maps to same driver_id then only either is effective (whichever is set first)?
> [Fiona] No,  the intention is that the session is initialised for all drivers, and so
> for all devices, using the same xform. So it should only be initialised to either
> compress or decompress.

[Shally] Ok. Then documentation can be updated to reflect this purpose.

> The intent being that an application can prepare an operation (stateless)
> independently of the driver & device it's targeting, and then choose where
> to send it, possibly based on which device is not busy.
> 
> 
[Shally] Ok. Then for now we can continue with same design approach as in cryptodev. Will propose any changes later on a need basis.
Also, I may have few questions on session API implementation in rte_compressdev.c. Will ask them in API RFC.

> >
> > Is this understanding correct?
> >
> > >
> > > > B. Notion of burst operations in compression API
> > > >  =======================================
> > > > struct rte_comp_op defines compression/decompression operational
> > > parameter and makes up one single
> > > > element of burst. This is both an input/output parameter.
> > > > PMD gets source, destination and checksum information at input and
> > > updated it with bytes consumed and
> > > > produced at output.
> > > [Fiona] Agreed
> > >
> > > > Once enqueued for processing, rte_comp_op *cannot be reused* until
> its
> > > status is set to
> > > > RTE_COMP_OP_FAILURE or RTE_COMP_OP_STATUS_SUCCESS.
> > > [Fiona] cannot be used until its status is set to any value other than
> > > RTE_COMP_OP_NOT_PROCESSED
> >
> > [Shally] How user will know that status is NOT_PROCESSED after ops are
> enqueued?
> > I assume only way to check enqueued ops status is dequeue_burst() and
> PMD put an op into completion
> > queue for dequeue *only when* it is completed with
> Pass/Fail/Out_of_space condition *not* when it's in
> > progress (equivalent of RTE_COMP_OP_NOT_PROCESSED).
> >
> > Am I missing anything here?
> [Fiona]  Correct. PMD should only return an op in the dequeue once it's
> processed.
> 
> >
> > >
> > >
> > > > C. Session and rte_comp_op
> > > >  =======================
> > > > Every operation in a burst is tied to a Session. More to cover on this
> under
> > > Stateless Vs Stateful section.
> > > [Fiona] Agreed. I would add that each operation in a burst may be
> attached
> > > to a different session.
> > >
> > >
> > > > D. Stateless Vs Stateful
> > > > ===================
> > > > Compression API provide RTE_COMP_FF_STATEFUL feature flag for
> PMD
> > > to reflect its support for Stateful
> > > > operation.
> > > [Fiona] Agreed.
> > >
> > > >
> > > > D.1 Compression API Stateless operation
> > > > ------------------------------------------------------
> > > > A Stateless operation means all enqueued packets are independent of
> > > each other i.e. Each packet has
> > > > -              Their flush value is set to RTE_FLUSH_FULL or
> RTE_FLUSH_FINAL
> > > (required only on compression
> > > > side),
> > > > -              All-of the required input and sufficient large buffer size to store
> > > output i.e. OUT_OF_SPACE can
> > > > never occur (required during both compression and decompression)
> > > >
> > > > In such case, PMD initiates stateless processing and releases acquired
> > > resources after processing of current
> > > > operation is complete i.e. full input consumed and full output written.
> > > > Application can attach same or different session to each packet and can
> > > make consecutive enque_burst()
> > > > calls i.e. Following is relevant usage:
> > > >
> > > > enqueued = rte_comp_enque_burst (dev_id, qp_id, ops1, nb_ops);
> > > > enqueued = rte_comp_enque_burst(dev_id, qp_id, ops2, nb_ops);
> > > > enqueued = rte_comp_enque_burst(dev_id, qp_id, ops3, nb_ops);
> > > >
> > > > *Note – Every call has different ops array i.e.  same rte_comp_op array
> > > *cannot be reused* to queue next
> > > > batch of data until previous ones are completely processed.
> > > >
> > > > Also if multiple threads calls enqueue_burst() on same queue pair then
> it’s
> > > application onus to use proper
> > > > locking mechanism to ensure serialized enqueuing of operations.
> > > [Fiona] Agreed to above stateless description.
> > >
> > >
> > > > Please note any time output buffer ran out of space during write then
> > > operation will turn “Stateful”.  See
> > > > more on Stateful under respective section.
> > > [Fiona] Let's come back to this later. An alternative is that
> OUT_OF_SPACE is
> > > returned and the  application
> > > must treat as a fail and resubmit the operation with a larger destination
> > > buffer.
> >
> > [Shally] Then I propose to add a feature flag
> "FF_SUPPORT_OUT_OF_SPACE" per xform type for flexible
> > PMD design.
> > As there're devices which treat it as error on compression but not on
> decompression.
> > If it is not supported, then it should be treated as failure condition and app
> can resubmit operation.
> > if supported, behaviour *To-be-Defined* under stateful.
> [Fiona] Can you explain 'turn stateful' some more?
> If compressor runs out of space during stateless operation, either comp or
> decomp, and turns stateful, how would the app know? And what would be in
> status, consumed and produced?
> Could it return OUT_OF_SPACE, and if both consumed and produced == 0

[Shally] If consumed = produced == 0, then it's not OUT_OF_SPACE condition.

> then the whole op must be resubmitted with a bigger output buffer. But if
> consumed and produced > 0 then app could take the output and submit next
> op
> continuing from consumed+1.
>
 
[Shally] consumed and produced will *always* be > 0 in case of OUT_OF_SPACE.
OUT_OF_SPACE means output buffer exhausted while writing data into it and PMD may have more to write to it. So in such case, PMD should set
Produced = complete length of output buffer
Status = OUT_OF_SPACE
consume, following possibilities here:
1. consumed = complete length of src mbuf means PMD has read full input, OR
2. consumed = partial length of src mbuf means PMD has read partial input

On seeing this status, app should consume output and re-enqueue same op with empty output buffer and src = consumed+1.
Please note as per current proposal, app should call rte_compdev_enqueue_stream() version of API if it doesn't know output size beforehand.

> 
> > >
> > > >  Typical API(flow-wise) to setup for stateless operation:
> > > > 1. rte_comp_session *sess = rte_comp_session_create(rte_mempool
> > > *pool);
> > > > 2. rte_comp_session_init (int dev_id, rte_comp_session *sess,
> > > rte_comp_xform *xform, rte_mempool
> > > > *sess_pool);
> > > > 3. rte_comp_op_pool_create(rte_mempool ..)
> > > > 4. rte_comp_op_bulk_alloc (struct rte_mempool *mempool, struct
> > > rte_comp_op **ops, uint16_t
> > > > nb_ops);
> > > > 5. for every rte_comp_op in ops[],
> > > >     5.1 rte_comp_op_attach_session(rte_comp_op *op,
> rte_comp_session
> > > *sess);
> > > >     5.2 set up with src/dst buffer
> > > > 6. enq = rte_compdev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,
> > > struct rte_comp_op **ops, uint16_t
> > > > nb_ops);
> > > > 7. dqu = rte_compdev_dequeue_burst(dev_id, qp_id, ops, enq);
> > > > 8. repeat 7 while (dqu < enq) // Wait till all of enqueued are dequeued
> > > > 9. Repeat 5.2 for next batch of data
> > > > 10. rte_comp_session_clear () // only reset private data memory area
> and
> > > *not* the xform and devid
> > > > information. In case, you want to re-use session.
> > > > 11. rte_comp_session_free(ret_comp_sess *session)
> > > [Fiona] ok. This is one possible flow. There are variations possible
> > >  - Above assumes all ops are using the same session, this is not
> necessarily
> > > the case. E.g. there
> > >    could be a compression session and a decompression session and a
> burst of
> > > ops may contain both.
> >
> > [Shally] Agree but assume applicable only for stateless until we cover
> stateful
> >
> > >    In this case Step 9 would be Repeat 5.1 as well as 5.2
> > >  - Also it would not be necessary to wait until the full burst is dequeued
> > > before doing
> > >    another enqueue - though of course the ops would need to be
> managed so
> > > only
> > >    those finished with are reused, or multiple sets of ops could be
> allocated.
> >
> > [Shally] Agree
> >
> > >  - What do you mean by Step 10 comment? The session only has private
> data.
> > > It's up to the PMD to
> > >    store whatever it needs from the xform. I think session_clear should
> mean
> > > all the data is zeroed in the session.
> > >    If the session is to be re-used then nothing needs to be cleared. Each op
> is
> > > already re-using the session in your
> > >    flow above without clearing between ops.
> > >    BUT this only applies to stateless
> >
> > [Shally] This came from my previous notion of session. Now when I see it
> analogy to cryptodev, its purpose
> > is clear to me.
> > But then I propose to rename API to rte_compdev_sess_term() to make it
> self-explanatory.
> [Fiona] So I think term isn't very clear, I guess it's short for terminate?
> If so I can change to rte_compdev_sess_terminate()

[Shally] Yes by term I meant terminate. So rte_compdev_sess_terminate() is fine.

> However I think we may not need this at all if we go with your stream
> suggestion below.
> As there may be only immutable data in the session - like algo, level,
> checksum type, etc.
> In cryptodev the session data is also immutable, but as it holds a key it's
> important to clear
> it before allowing the session to be re-used. I'm not sure we'll have any data
> left in
> session which needs to be cleared, all the state data may be in the stream.
> 

[Shally] I would say we still need it to reverse operations done in sess_init() like free up device private data.
So, we should retain it.

> >
> > - for stateful we may need a different
> > > behaviour - to clear state data but
> > >    keep algo, level, Huffman-type, checksum-type. Let's discuss under
> stateful
> > > - this may need a new API.
> > >
> >
> > [Shally] Or PMD can internally reset its state once it process an op with
> FULL_FLUSH/FINISH. Will revisit it
> > under stateful.
> [Fiona] Yes, that could also be an option for stream.
> 
> 
> > > > D.1.2 Requirement for Stateless
> > > > -------------------------------------------
> > > > Since operation can complete out-of-order. There should be one (void
> > > *user) per rte_comp_op to enable
> > > > application to map dequeued op to enqueued op.
> > > [Fiona] In cryptodev there was an opaque_data field in the op - it was
> > > removed as the application can store any
> > > private data it needs following the op, as it creates the op pool and
> dictates
> > > the size. Do you think we need an
> > > explicit (void *user) or can we follow the same approach?
> > >
> >
> > [Shally] if priv_data in crypto_op_pool_create() is user data,  then we don't
> need explicit user *.
> > But then I propose variable name should be renamed to user_data
> indicating it is an app data opaque to
> > PMD.
> > And, __rte_comp_op_get_priv_data_size () should be changed to
> __rte_comp_op_get_user_data_size().
> [Fiona] Agreed.
> 
> 
> > > [Fiona] Out of time - I'll continue from here later in the week.
> > [Shally] Sure. Look forward to that.
> >
> > Thanks
> > Shally
> >
> > >
> > >
> > > > D.2 Compression API Stateful operation
> > > > ----------------------------------------------------------
> > > >  A Stateful operation means following conditions:
> > > > - API ran into out_of_space situation during processing of input.
> Example,
> > > stateless compressed stream
> > > > fed fully to decompressor but output buffer is not large enough to hold
> > > output.
> > > > - API waiting for more input to produce output. Example, stateless
> > > compressed stream fed partially to
> > > > decompressor.
> [Fiona] I agree this is stateful if more input is to follow. But don’t understand
> the
> stateless ref. here. It's not relevant whether compression was done
> statefully
> or statelessly, if only part of the data is fed to the decompressor then it
> should
> be fed to a stateful session.
> 

[Shally] Agree to what you said. This is regardless of stateful/stateless.

> 
> > > > - API is dependent on previous operation for further
> > > compression/decompression
> > > >
> > > > In case of either one or all of the above conditions PMD is required to
> > > maintain context of operations
> > > > across enque_burst() calls, until a packet with  RTE_FLUSH_FULL/FINAL
> and
> > > sufficient input/output buffers
> > > > is received and processed.
> > > >
> > > > D.2.1 Compression API requirement for Stateful
> > > > ---------------------------------------------------------------
> > > >
> > > > D.2.1.1 Sliding Window Size
> > > > ------------------------------------
> > > > Maximum length of Sliding Window in bytes. Previous data lookup will
> be
> > > performed up to this length. To
> > > > be added as algorithm capability parameter and set by PMD.
> [Fiona] Agreed. 32k default.
> To be set on session xform for both compression and decompression?

[Shally] Yes for both compression and decompression.

> Is there a subset of specific values you'd like supported on the API?
>  (rather than all to minimise test cases)

[Shally] I would suggest to stay with current approach of providing max. 
Though it may increase test efforts but that way it will be more conformant to zlib/deflate RFC and extensible to meet different app need.

> 
> 
> > > >
> > > > D.2.1.2 Stateful operation state maintenance
> > > >  -------------------------------------------------------------
> > > > This section starts with description of our understanding about
> > > compression API support for stateful.
> > > > Depending upon understanding build upon these concepts, we will
> identify
> > > required data structure/param
> > > > to maintain in-progress operation context by PMD.
> > > >
> > > > For stateful compression, batch of dependent packets starts at a packet
> > > having
> > > > RTE_NO_FLUSH/RTE_SYNC_FLUSH flush value and end at packet having
> > > RTE_FULL_FLUSH/FINAL_FLUSH.
> > > > i.e. array of operations will carry structure like this:
> > > >
> > > > ------------------------------------------------------------------------------------
> > > > |op1.no_flush | op2.no_flush | op3.no_flush | op4.full_flush|
> > > > ------------------------------------------------------------------------------------
> > > >
> [Fiona] I think it needs to be more constrained than your examples below.
> Only 1 operation from a stream can be in a burst. As each operation
> in a stateful stream must complete, as next operation needs state and
> history
> of previous operation to be complete before it can be processed.
> And if one failed, e.g. due to OUT_OF_SPACE, this should affect
> the following operation in the same stream.
> Worst case this means bursts of 1. Burst can be >1 if there are multiple
> independent streams with available data for processing. Or if there is
> data available which can be statelessly processed.
> 
> If there are multiple buffers available from a stream , then instead they can
> be linked together in an mbuf chain sent in a single operation.
> 
> To handle the sequences below would mean the PMD
> would need to store ops sending one at a time to be processed.
> 
> As this is significantly different from what you describe below, I'll wait for
> further feedback
> before continuing.

[Shally] I concur with your thoughts. And these're are not significantly different from the concept presented below.

Yes as you mentioned, even for burst_size>1 PMD will have to serialize each op internally i.e. 
It has to wait for previous to finish before putting next for processing which is 
as good as application making serialised call passing one op at-a-time or if
stream consists of multiple buffers, making their scatter-gather list and 
then enqueue it as one op at a time which is more efficient and ideal usage.
However in order to allow extensibility, I didn't mention limitation on burst_size. 
Because If PMD doesn't support burst_size > 1 it can always return nb_enqueued = 1, in which case
app can enqueue next however with condition it should wait for previous to complete
before making next enqueue call.

So, if we take simple example to compress 2k of data with src mbuf size = 1k. 
Then with burst_size=1, expected call flow would be(this is just one flow, other variations are also possible suchas making chain of 1k buffers and pass whole data in one go):

1. fill 1st 1k chunk of data in op.msrc
2.enqueue_stream (..., |op.flush = no_flush|, 1, ptr_stream);
3.dequeue_burst(|op|,1);
4.refill next 1k chunk in op.msrc
5.enqueue_stream(...,|op.flush = full_flush|, 1 , ptr_stream);
6.dequeue_burst(|op|, 1);
7.end

So, I don’t see much of a change in API call flow from here to design presented below except nb_ops = 1 in each call.
However I am assuming that op structure would still be same for stateful processing i.e. it would start with op.flush value = NO/SYNC_FLUSH and end at op with flush value = FULL FLUSH.
Are we on same page here? 

Thanks
Shally

> 
> But in principal I agree with the idea of a stream structure to hold the state
> data
> needed for a stateful flow. Originally I'd assumed this data would be held in
> the session
> but I think it's better to not have a 1:1 mapping between stream and session
> as this
> would limit a session unnecessarily. So there can be many streams in a single
> session.
> 
> 
> > > > For sake of simplicity, we will use term "stream" to identify such related
> set
> > > of operation in following
> > > > description.
> > > >
> > > > Stream processing impose following limitations on usage of
> enque_burst()
> > > API
> > > > -              All dependent packets in a stream should carry same session
> > > > -              if stream is broken into multiple enqueue_burst() call, then
> next
> > > enqueue_burst() cannot be called
> > > > until previous one has fully processed. I.E.
> > > >
> > > >                Consider for example, a stream with ops1 ..ops7,  This is *not*
> > > allowed
> > > >
> > > >                                        ----------------------------------------------------------------
> ----
> > > --------------
> > > >                 enque_burst(|op1.no_flush | op2.no_flush | op3.no_flush |
> > > op4.no_flush|)
> > > >                                        ----------------------------------------------------------------
> ----
> > > --------------
> > > >
> > > >                                        ----------------------------------------------------------------
> > > >                enque_burst(|op5.no_flush | op6.no_flush | op7.flush_final |)
> > > >                                         ---------------------------------------------------------------
> -
> > > >
> > > >               This *is* allowed
> > > >                                        ----------------------------------------------------------------
> ----
> > > --------------
> > > >                enque_burst(|op1.no_flush | op2.no_flush | op3.no_flush |
> > > op4.no_flush|)
> > > >                                        ----------------------------------------------------------------
> ----
> > > --------------
> > > >
> > > >                 deque_burst(ops1 ..ops4)
> > > >
> > > >                                        ----------------------------------------------------------------
> > > >                enque_burst(|op5.no_flush | op6.no_flush | op7.flush_final |)
> > > >                                         ---------------------------------------------------------------
> -
> > > >
> > > > -              A single enque_burst() can carry only one stream. I.E. This is
> *not*
> > > allowed
> > > >
> > > >                                       ----------------------------------------------------------------
> -----
> > > ------------------------------------
> > > >               enque_burst (|op1.no_flush | op2.no_flush | op3.flush_final |
> > > op4.no_flush | op5.no_flush |)
> > > >                                        ----------------------------------------------------------------
> ----
> > > -------------------------------------
> > > >
> > > > If a stream is broken in to several enqueue_burst() calls, then compress
> > > API need to maintain operational
> > > > state between calls. For this, concept of rte_comp_stream is enabled in
> to
> > > compression API.
> > > > Here’re the proposed changes to existing design:
> > > >
> > > > 1. Add rte_comp_op_type
> > > > ........................................
> > > > enum rte_comp_op_type {
> > > > RTE_COMP_OP_STATELESS,
> > > > RTE_COMP_OP_STATEFUL
> > > > }
> > > >
> > > > 2. Add new data type rte_comp_stream to maintain stream state
> > > > ........................................................................................................
> > > > rte_comp_stream is an opaque data structure to application which is
> > > exchanged back and forth between
> > > > application and PMD during stateful compression/decompression.
> > > > It should be allocated per stream AND before beginning of stateful
> > > operation. If stream is broken into
> > > > multiple enqueue_burst() then each
> > > > respective enqueue_burst() must carry same rte_comp_stream
> pointer. It
> > > is mandatory input for stateful
> > > > operations.
> > > > rte_comp_stream can be cleared and reused via compression API
> > > rte_comp_stream_clear() and free via
> > > > rte_comp_stream_free(). Clear/free should not be called when it is in
> use.
> > > >
> > > > This enables sharing of a session by multiple threads handling different
> > > streams as each bulk ops carry its
> > > > own context. This can also be used by PMD to handle OUT_OF_SPACE
> > > situation.
> > > >
> > > > 3. Add stream allocate, clear and free API
> > > > ...................................................................
> > > > 3.1. rte_comp_op_stream_alloc(rte_mempool *pool,
> rte_comp_op_type
> > > type, rte_comp_stream
> > > > **stream);
> > > > 3.2. rte_comp_op_stream_clear(rte_comp_stream *stream); // in this
> > > case stream will be useable for new
> > > > stateful batch
> > > > 3.3. rte_comp_op_stream_free(rte_comp_stream *stream); // to free
> > > context
> > > >
> > > > 4. Add new API rte_compdev_enqueue_stream()
> > > > ...............................................................................
> > > >  static inline uint16_t rte_compdev_enqueue_stream(uint8_t dev_id,
> > > >                                                  uint16_t qp_id,
> > > >                                                  struct rte_comp_op **ops,
> > > >                                                  uint16_t nb_ops,
> > > >                                                 rte_comp_stream *stream); //to be passed
> with
> > > each call
> > > >
> > > > Application should call this API to process dependent set of data OR
> when
> > > output buffer size is unknown.
> > > >
> > > > rte_comp_op_pool_create() should create mempool large enough to
> > > accommodate operational state
> > > > (maintained by rte_comp_stream) based on rte_comp_op_type. Since
> > > rte_comp_stream would be
> > > > maintained by PMD, thus allocating it from PMD managed pool offers
> > > performance gains.
> > > >
> > > > API flow: rte_comp_op_pool_create() -→ rte_comp_op_bulk_alloc() --
> ->
> > > rte_comp_op_stream_alloc() →
> > > > enque_stream(..ops, .., stream)
> > > >
> > > > D.2.1.3 History buffer
> > > > -----------------------------
> > > > Will be maintained by PMD with in rte_comp_stream
> > > >
> > > > Thanks
> > > > Shally



More information about the dev mailing list