[dpdk-dev] [PATCH RFC] librte_reorder: new reorder library

[Pattan, Reshma]

> -----Original Message-----
> From: Neil Horman [mailto:nhorman at tuxdriver.com]
> Sent: Thursday, October 9, 2014 12:37 PM
> To: Pattan, Reshma
> Cc: dev at dpdk.org
> Subject: Re: [dpdk-dev] [PATCH RFC] librte_reorder: new reorder library
> 
> On Thu, Oct 09, 2014 at 10:27:55AM +0000, Pattan, Reshma wrote:
> >
> >
> > > -----Original Message-----
> > > From: Neil Horman [mailto:nhorman at tuxdriver.com]
> > > Sent: Wednesday, October 8, 2014 8:16 PM
> > > To: Pattan, Reshma
> > > Cc: dev at dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH RFC] librte_reorder: new reorder
> > > library
> > >
> > > On Wed, Oct 08, 2014 at 02:11:34PM +0000, Pattan, Reshma wrote:
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Neil Horman [mailto:nhorman at tuxdriver.com]
> > > > > Sent: Tuesday, October 7, 2014 12:22 PM
> > > > > To: Pattan, Reshma
> > > > > Cc: dev at dpdk.org
> > > > > Subject: Re: [dpdk-dev] [PATCH RFC] librte_reorder: new reorder
> > > > > library
> > > > >
> > > > > On Tue, Oct 07, 2014 at 09:33:06AM +0000, Pattan, Reshma wrote:
> > > > > > Hi All,
> > > > > >
> > > > > > I am planning  to implement packet reorder library. Details
> > > > > > are as below,
> > > > > please go through them and provide the comments.
> > > > > >
> > > > > > Requirement:
> > > > > >                To reorder out of ordered packets that are
> > > > > > received from different
> > > > > cores.
> > > > > >
> > > > > > Usage:
> > > > > > To be used along with distributor library. Next version of
> > > > > > distributor are
> > > > > planned to distribute incoming packets to all worker cores
> > > > > irrespective of the flow type.
> > > > > > In this case to ensure in order delivery of the packets at
> > > > > > output side reorder
> > > > > library will used by the tx end.
> > > > > >
> > > > > > Assumption:
> > > > > > All input packets will be marked with sequence number in seqn
> > > > > > field of mbuf,
> > > > > this will be the reference for reordering at the tx end.
> > > > > > Sequence number will be of type uint32_t. New sequence number
> > > > > > field seqn
> > > > > will be added to mbuf structure.
> > > > > >
> > > > > > Design:
> > > > > > a)There will be reorder buffer(circular buffer) structure
> > > > > > maintained in reorder
> > > > > library to store reordered packets and other details of buffer
> > > > > like head to drain the packet from, min sequence number and other
> details.
> > > > > >                b)Library will provide insert and drain
> > > > > > functions to reorder and fetch
> > > > > out the reordered packets respectively.
> > > > > > c)Users of library should pass the packets to insert functions for
> reordering.
> > > > > >
> > > > > > Insertion logic:
> > > > > > Sequence number of current packet will be used to calculate
> > > > > > offset in reorder
> > > > > buffer and write packet to the location  in the reorder buffer
> > > > > corresponding to offset.
> > > > > >                              Offset is calculated as
> > > > > > difference of current packet  sequence
> > > > > number and sequence number associated with  reorder buffer.
> > > > > >
> > > > > > During sequence number wrapping or wrapping over of reorder
> > > > > > buffer size,
> > > > > before inserting the new packet we should move offset number of
> > > > > packets to other buffer called overflow buffer and advance the
> > > > > head of reorder buffer by "offset-reorder buffer size" and insert the new
> packet.
> > > > > >
> > > > > > Insert function:
> > > > > > int rte_reorder_insert(struct rte_reorder_buffer *buffer,
> > > > > > struct rte_mbuf *mbuf);
> > > > > > Note: Other insert function is also under plan to insert burst of packets.
> > > > > >
> > > > > >                Reorder buffer:
> > > > > > struct rte_reorder_buffer {
> > > > > >         unsigned int size;      /* The size (number of entries) of the buffer.
> */
> > > > > >         unsigned int mask;      /* Mask (size - 1) of the buffer */
> > > > > >         unsigned int head;      /* Current head of buffer */
> > > > > >         uint32_t min_seqn;      /* latest sequence number associated with
> > > buffer
> > > > > */
> > > > > >         struct rte_mbuf *entries[MAX_REORDER_BUFFER_SIZE]; /*
> > > > > > buffer to hold reordered mbufs */ };
> > > > > >
> > > > > > d)Users can fetch out the reordered packets by drain function
> > > > > > provided by
> > > > > library. Users must pass the mbuf array , drain function should
> > > > > fill passed mbuff array  with the reordered buffer packets.
> > > > > > During drain operation, overflow buffer  packets will be
> > > > > > fetched out first and
> > > > > then reorder buffer.
> > > > > >
> > > > > > Drain function:
> > > > > >                int rte_reorder_drain(struct rte_reorder_buffer
> > > > > > *buffer, struct rte_mbuf **mbufs)
> > > > > >
> > > > > > Thanks,
> > > > > > Reshma
> > > > > >
> > > > >
> > > > > This seems reasonable, but why not integrate it with the
> > > > > distributor library rather than create a separate library for
> > > > > it?  It seems as though the distributor library is a
> > > > > pre-requisite for this libraries use anyway, as otherwise there
> > > > > will not be anything to reorder Neil
> > > > >
> > > >
> > > > Hi  Neil,
> > > >
> > > > Reorder library should be standalone , as there are many ways that
> > > > can cause out of ordering of packets, I just mentioned future
> > > > packet distributor
> > > enhancements  as one of the example for out of ordering.
> > > > Other ways like, users can directly distribute the packets to
> > > > different cores via
> > > rings and that causes packet out of ordering as well.
> > > > So, keeping reorder library standalone would be good to work with
> > > > all packet
> > > distribution ways.
> > > >
> > >
> > >
> > > Hmm, ok, that seems reasonable.
> > >
> > > Just out of curiosity, where do you intend to inject the packet
> > > sequence number for this library?
> > >
> > Sequence number marking can be done in  either of these places  1)PMD rx
> side 2)  packet distributor process  or 3) in application itself.
> >
> Thanks.  So, that was the part of this that was getting to me. For this to work,
> you're going to have to mark frames externally to the re-order library (i.e.
> You're proposed library needs to rely on the application or the pmd to do proper
> marking of frames to reorder properly).  Is it tennable in your mind to require
> that?
> 

For the users of distributor sequence number marking will be automatic. 
We are also considering ways to make sequence number marking automatic in PMD RX  but at cost of performance hit. This approach will be optional  but not mandatory,  i.e.  user configurable.

Thanks,
Reshma

> Neil
> 
> > Thanks,
> > Reshma
> >
> > > Neil
> >
> >