[dpdk-dev] Order of system brought up affects throughput with qos_sched app

[Dumitrescu, Cristian]

Hi Wei,

Here is another hypothesis for you to consider: if the size of your token buckets (used to store subport and pipe credits) is big (and it actually is set big in the default config file of the app), then when no packets are received for a long while (which is the case when you start the app first and the traffic gen later), the token buckets are continuously replenished (with nothing consumed) until they become full; when packets start to arrive, the token buckets are full and it can take a long time (might be minutes or even hours, depending on how big your buckets are) until they come down to their normal values (this time can actually be comuted/estimated).

If this is what happens in your case, lowering the size of your buckets will help.

Regards,
Cristian

> -----Original Message-----
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Wei Shen
> Sent: Wednesday, September 9, 2015 9:39 PM
> To: dev at dpdk.org
> Subject: [dpdk-dev] Order of system brought up affects throughput with
> qos_sched app
> 
> Hi all,
> I ran into problems with qos_sched with different order of system brought
> up. I can bring up the system in two ways:
> 1. Start traffic gen first. Then start qos_sched.2. Start qos_sched first. Then
> start traffic gen.
> With 256K pipes and 64 queue size, 128B packet size, I got ~4Gbps with order
> #1. While I got 10G with order #2.
> qos_sched command stats showed that ~59% packets got dropped in RX
> (rte_ring_enqueue).
> Plus, with #1, if I restart the traffic gen later, I would regain 10Gbps
> throughput, which suggests that this is not an initialization issue but runtime
> behavior.
> I also tried to assign qos_sched on different cores and got the same result.
> I suspect that there is some rte_ring bugs when connecting two cores and
> one core started enqueuing before another core is ready for dequeue.
> Have you experienced the same issue? Appreciate your help.
> 
> Wei Shen.---------------------------------------------------------------------------------
> ---------------------------------------------------My system spec is:Intel(R) Xeon(R)
> CPU E5-2699 v3 @ 2.30GHz15 * 1G hugepages
> qos_sched argument: ./build/app/qos_sched -c 1c0002 -n 4 -- --pfc
> "0,1,20,18,19" --cfg profile.cfg
> profile.cfg:[port]frame overhead = 20number of subports per port =
> 1number of pipes per subport = 262144queue sizes = 64 64 64 64
> 
> ; Subport configuration[subport 0]tb rate = 1250000000           ; Bytes per
> secondtb size = 1000000                 ; Bytes
> tc 0 rate = 1250000000         ; Bytes per secondtc 1 rate = 1250000000         ;
> Bytes per secondtc 2 rate = 1250000000         ; Bytes per secondtc 3 rate =
> 1250000000         ; Bytes per secondtc period = 10                        ; Milliseconds
> pipe 0-262143 = 0                ; These pipes are configured with pipe profile 0
> ; Pipe configuration[pipe profile 0]tb rate = 1250000000           ; Bytes per
> secondtb size = 1000000                 ; Bytes
> tc 0 rate = 1250000000         ; Bytes per secondtc 1 rate = 1250000000         ;
> Bytes per secondtc 2 rate = 1250000000         ; Bytes per secondtc 3 rate =
> 1250000000         ; Bytes per secondtc period = 10                        ; Milliseconds
> tc 3 oversubscription weight = 1
> tc 0 wrr weights = 1 1 1 1tc 1 wrr weights = 1 1 1 1tc 2 wrr weights = 1 1 1 1tc 3
> wrr weights = 1 1 1 1