[dpdk-dev] [PATCH v2 09/28] sched: update pkt read and write API

[Dumitrescu, Cristian]

> -----Original Message-----
> From: Singh, Jasvinder
> Sent: Tuesday, July 2, 2019 10:05 PM
> To: Dumitrescu, Cristian <cristian.dumitrescu at intel.com>; dev at dpdk.org
> Cc: Tovar, AbrahamX <abrahamx.tovar at intel.com>; Krakowiak, LukaszX
> <lukaszx.krakowiak at intel.com>
> Subject: RE: [PATCH v2 09/28] sched: update pkt read and write API
> 
> 
> 
> > -----Original Message-----
> > From: Dumitrescu, Cristian
> > Sent: Tuesday, July 2, 2019 12:25 AM
> > To: Singh, Jasvinder <jasvinder.singh at intel.com>; dev at dpdk.org
> > Cc: Tovar, AbrahamX <abrahamx.tovar at intel.com>; Krakowiak, LukaszX
> > <lukaszx.krakowiak at intel.com>
> > Subject: RE: [PATCH v2 09/28] sched: update pkt read and write API
> >
> >
> >
> > > -----Original Message-----
> > > From: Singh, Jasvinder
> > > Sent: Tuesday, June 25, 2019 4:32 PM
> > > To: dev at dpdk.org
> > > Cc: Dumitrescu, Cristian <cristian.dumitrescu at intel.com>; Tovar,
> > > AbrahamX <abrahamx.tovar at intel.com>; Krakowiak, LukaszX
> > > <lukaszx.krakowiak at intel.com>
> > > Subject: [PATCH v2 09/28] sched: update pkt read and write API
> > >
> > > Update run time packet read and write api implementation to allow
> > > configuration flexiblity for pipe traffic classes and queues, and
> > > subport level configuration of the pipe parameters.
> > >
> > > Signed-off-by: Jasvinder Singh <jasvinder.singh at intel.com>
> > > Signed-off-by: Abraham Tovar <abrahamx.tovar at intel.com>
> > > Signed-off-by: Lukasz Krakowiak <lukaszx.krakowiak at intel.com>
> > > ---
> > >  lib/librte_sched/rte_sched.c | 32 +++++++++++++++++---------------
> > > lib/librte_sched/rte_sched.h |  8 ++++----
> > >  2 files changed, 21 insertions(+), 19 deletions(-)
> > >
> > > diff --git a/lib/librte_sched/rte_sched.c
> > > b/lib/librte_sched/rte_sched.c index 1999bbfa3..cd82fd918 100644
> > > --- a/lib/librte_sched/rte_sched.c
> > > +++ b/lib/librte_sched/rte_sched.c
> > > @@ -1433,17 +1433,15 @@ rte_sched_port_pipe_profile_add(struct
> > > rte_sched_port *port,
> > >
> > >  static inline uint32_t
> > >  rte_sched_port_qindex(struct rte_sched_port *port,
> > > +	struct rte_sched_subport *s,
> > >  	uint32_t subport,
> > >  	uint32_t pipe,
> > > -	uint32_t traffic_class,
> > >  	uint32_t queue)
> > >  {
> > >  	return ((subport & (port->n_subports_per_port - 1)) <<
> > > -			(port->n_pipes_per_subport_log2 + 4)) |
> > > -			((pipe & (port->n_pipes_per_subport - 1)) << 4) |
> > > -			((traffic_class &
> > > -			    (RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE - 1)) <<
> > > 2) |
> > > -			(queue &
> > > (RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS - 1));
> > > +			(port->max_subport_pipes_log2 + 4)) |
> > > +			((pipe & (s->n_subport_pipes - 1)) << 4) |
> > > +			(queue & (RTE_SCHED_QUEUES_PER_PIPE - 1));
> > >  }
> > >
> >
> > This function contains a critical bug: this patchset proposes that the number
> of
> > pipes per subport is configurable independently for each subport; in other
> > words, each subport can be configured with a different number of pipes.
> > Therefore, the above logic is broken, as it assumes all subports have the
> same
> > number of pipes. There is no longer possible to compute port-
> > >max_subport_pipes_log2. Correct?
> 
> Yes, you are right. I didn't realize this issue.
> 
> >
> > We might need to rethink the design solution for the per-subport
> independent
> > configuration.
> 
> One option to get around this is by computing  start_queue_offset for each
> subport and store that in rte_sched_subport data structure during subport
> configuration.
> 

Yes, it can be done, this is probably the only way to get it done, but it would severely impact the performance, as in order to determine the subport ID you'd have to iterate through the list of subports to search where this queue ID fits.

> During run time, when writing pkt metadata;  add that offset to calculate
> queue index ;
> 	subport->start_queue_offset+((pipe & (s->n_pipes_per_subport -
> 1)) << 4) | (queue & (RTE_SCHED_QUEUES_PER_PIPE - 1));

> At the same time,  subport id can be written to reserved field of the mbuf-
> >hash.sched

I am afraid we cannot write subport ID into mbuf->sched, as subport ID is not generic, it is only specific to librte_sched feature set. This will pollute the generic mbuf->sched with librte_sched implementation details.

> 
> During packet read, offset value is retrieved from subport_id (from reserved
> field). By subtracting offset from the qindex,
> pipe, tc and queue id can determined from the remaining value.
> 
> This will allow contiguous value of the queue id at the port level.
> 
> > We also need to make sure we test this library with multiple subports per
> port,
> > with each subport having different number of pipes. Need to do the basic
> uni
> > test proposed earlier to trace the packet through the scheduler hierarchy
> up to
> > the packet queue.
> 
> Yes, will add unit test for this case.
> >
> > >  void
> > > @@ -1453,9 +1451,9 @@ rte_sched_port_pkt_write(struct
> rte_sched_port
> > > *port,
> > >  			 uint32_t traffic_class,
> > >  			 uint32_t queue, enum rte_color color)  {
> > > -	uint32_t queue_id = rte_sched_port_qindex(port, subport, pipe,
> > > -			traffic_class, queue);
> > > -	rte_mbuf_sched_set(pkt, queue_id, traffic_class, (uint8_t)color);
> > > +	struct rte_sched_subport *s = port->subports[subport];
> > > +	uint32_t qindex = rte_sched_port_qindex(port, s, subport, pipe,
> > > queue);
> > > +	rte_mbuf_sched_set(pkt, qindex, traffic_class, (uint8_t)color);
> > >  }
> > >
> >
> > Same comment here.
> >
> > >  void
> > > @@ -1464,13 +1462,17 @@ rte_sched_port_pkt_read_tree_path(struct
> > > rte_sched_port *port,
> > >  				  uint32_t *subport, uint32_t *pipe,
> > >  				  uint32_t *traffic_class, uint32_t *queue)  {
> > > -	uint32_t queue_id = rte_mbuf_sched_queue_get(pkt);
> > > +	struct rte_sched_subport *s;
> > > +	uint32_t qindex = rte_mbuf_sched_queue_get(pkt);
> > > +	uint32_t tc_id = rte_mbuf_sched_traffic_class_get(pkt);
> > > +
> > > +	*subport = (qindex >> (port->max_subport_pipes_log2 + 4)) &
> > > +		(port->n_subports_per_port - 1);
> > >
> > > -	*subport = queue_id >> (port->n_pipes_per_subport_log2 + 4);
> > > -	*pipe = (queue_id >> 4) & (port->n_pipes_per_subport - 1);
> > > -	*traffic_class = (queue_id >> 2) &
> > > -				(RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE -
> > > 1);
> > > -	*queue = queue_id & (RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS -
> > > 1);
> > > +	s = port->subports[*subport];
> > > +	*pipe = (qindex >> 4) & (s->n_subport_pipes - 1);
> > > +	*traffic_class = tc_id;
> > > +	*queue = qindex & (RTE_SCHED_QUEUES_PER_PIPE - 1);
> > >  }
> > >
> >
> > Same comment here.
> >
> > >  enum rte_color
> > > diff --git a/lib/librte_sched/rte_sched.h
> > > b/lib/librte_sched/rte_sched.h index 121e1f669..6a6ea84aa 100644
> > > --- a/lib/librte_sched/rte_sched.h
> > > +++ b/lib/librte_sched/rte_sched.h
> > > @@ -421,9 +421,9 @@ rte_sched_queue_read_stats(struct
> rte_sched_port
> > > *port,
> > >   * @param pipe
> > >   *   Pipe ID within subport
> > >   * @param traffic_class
> > > - *   Traffic class ID within pipe (0 .. 3)
> > > + *   Traffic class ID within pipe (0 .. 8)
> > >   * @param queue
> > > - *   Queue ID within pipe traffic class (0 .. 3)
> > > + *   Queue ID within pipe traffic class (0 .. 15)
> > >   * @param color
> > >   *   Packet color set
> > >   */
> > > @@ -448,9 +448,9 @@ rte_sched_port_pkt_write(struct rte_sched_port
> > > *port,
> > >   * @param pipe
> > >   *   Pipe ID within subport
> > >   * @param traffic_class
> > > - *   Traffic class ID within pipe (0 .. 3)
> > > + *   Traffic class ID within pipe (0 .. 8)
> > >   * @param queue
> > > - *   Queue ID within pipe traffic class (0 .. 3)
> > > + *   Queue ID within pipe traffic class (0 .. 15)
> > >   *
> > >   */
> > >  void
> > > --
> > > 2.21.0