[dpdk-dev] RFC: compressdev: append dest data in PMDs instead of in application

[Trahe, Fiona]

Proposal:
In compressdev, move the responsibility for appending data in the destination
mbuf from the application to the PMD.

Why:
Compression operations are all out-of-place and the output size is unknown.
Source and destination mbufs are passed to the PMDs.
There's no problem with the src, but there is with the destination.
The application allocates the dest mbuf from the pool, guesses how much of
the buffer the PMD will write with output and calls rte_pktmbuf_append() for this amount. 
No data is actually copied into the dest mbuf by the application.
 
The PMD writes output data to the destination mbuf, and returns the amount written in the 
op.produced parameter.

Throughout this the mbuf is not consistent with expectations, i.e. a call to
rte_pktmbuf_data_len() will NOT return the actual amount of data in the buffer. A call to
rte_pktmbuf_append() will NOT add space at end of existing data. rte_pktmbuf_tailroom()
 will not return how much space is available at the end of the data.
Also, some PMDs, e.g. ISA-L, need scratch space at end of the output buffer for checksum
calculation. So though the appl has appended a specific amount in the expectation that it
can be used for compressed data, the PMD needs to reduce this by 4 bytes to reserve
space for the checksum - even though there may be space to append another 4bytes.

It seems more logical that the PMD should take responsibility for appending.
I.e. the application should pass in an empty mbuf, the PMD uses the rte_pktmbuf_tailroom()
to know what space is available, uses this space as it needs and appends the output data to
match the actual amount of data it writes.
This method caters for an mbuf already containing some data, in this case the PMD will
append the output AFTER the data already in the mbuf.
It also needs to cater for SGL aka chained_mbuf case, though I'd propose in this case only
the first mbuf in the chain is allowed to already contain data, the rest must be empty.
An application can adjust a chain to satisfy this condition.

Code impacts:
 * rte_comp_op.dst.offset should be deprecated.
 * comments on the m_dst would be changed to describe this usage
 * applications should alloc destination mbuf(s) from a pool, but not append.
 * PMDs should use append() to correctly reflect the amount of data returned.

This turns out to be more complex than expected for SGLs as rte_mbuf chains DON'T support
empty mbuf chains, i.e. several macros assume there's only tailroom available in the last 
segment of a chain. So essentially if an application chains a bunch of empty mbufs
together the only tailroom available is the buf_len of the last mbuf and the space in
all the other mbufs is "lost".
We've investigated several ways around this, I think either options 1 or 2 would be ok, but
am not keen on option 3. I'm looking for feedback please.

Option1: create an rte_comp_mbuf, identical to rte_mbuf - probably just cast to it - with its own set of
   macros.  Most of these wrap existing mbuf macros, those that need to cater for the empty chain case.
Option2: pass in a pool on the API instead and the PMD allocs dest mbufs as needed and chains them.
   Storage customers expect to use external buffers attached to mbufs so this may not suit their use-case.
   Although it may be an option if all mbufs in the pool are pre-attached to external buffers.
Option3: appl does as today. PMD trims space not used. Would need changes or additions to mbuf macros
   so it could trim from more than just the last segment. PMD would need to free unused segments. 
   I'm not keen on this as doing the work in 2 places and there's potential
   to leak mbufs here as allocating them in API and freeing in PMD.

Regards,
Fiona