[dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hanoch Haim (hhaim)
hhaim at cisco.com
Wed Dec 6 12:21:50 CET 2017
Hi Beilei,
Thanks for looking into this.
I'm testing it with TRex (tx->rx) and not (rx->tx) in a bit different configuration and there are packets with ~40usec latency (Tx/Rx) without my patch
So I assume the latency comes from a different place due to this change (INTR)
TRex command:
$sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -c 1
I'm testing it in loopback and in this configuration there are two Tx queues and 2 Rx queues
RSS is not configured all Rx traffic goes to Rx0
Core0
--------
Tx0 - TCP flows
Rx0 - TCP flows
Core1
--------
Tx1 - latency flows (ICMP) - low rate 1KPPS
Rx1 - latency flows (ICMP) - match a rule of TOS/TTL to redirect to this Rx1 queue.
Qos configuration
------------------------
ETS, Tx1 is configured as low latency/high priority
Throughput of Tx0/Rx0 is ~10Gb/average packet size =~700B
The latency is measured betwean Tx1 to Rx1
Thanks,
Hanoh
-----Original Message-----
From: Xing, Beilei [mailto:beilei.xing at intel.com]
Sent: Wednesday, December 06, 2017 1:00 PM
To: Wu, Jingjing; Hanoch Haim (hhaim)
Cc: dev at dpdk.org
Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF
Hi,
I tested Rx latency with testpmd and ixia, found that the interval configuration works on X710.
With the default configuration, the latency is about 32us.
When RTE_LIBRTE_I40E_ITR_INTERVAL=0, the max latency is < 8us.
When RTE_LIBRTE_I40E_ITR_INTERVAL=8160, the max latency is about 8ms.
My test steps:
1. connect a X710 port with ixia, and bind the port to igb_uio 2. run testpmd and start io forwding 3. send a burst (5 packets) with ixia to X710 port 4. check the latency on ixia.
Best Regards,
Beilei
> -----Original Message-----
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Wu, Jingjing
> Sent: Friday, December 1, 2017 1:47 AM
> To: Hanoch Haim (hhaim) <hhaim at cisco.com>
> Cc: dev at dpdk.org
> Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi, Hanoch
>
> Thanks a lot for the trying. Are you using igb_uio to bind the device?
>
> I guess it would be because that fix is not complete, the overlap with
> Rx interrupt mode is not considered. We will look into it.
> And it would be great if you can have a try on vfio_pci cases.
>
> Thanks
> Jingjing
>
> From: Hanoch Haim (hhaim) [mailto:hhaim at cisco.com]
> Sent: Friday, December 1, 2017 12:48 AM
> To: Wu, Jingjing <jingjing.wu at intel.com>
> Cc: dev at dpdk.org; Hanoch Haim (hhaim) <hhaim at cisco.com>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi Jingjing,
> I did that and see the results, It does not work as expected
>
> TRex command:
> $sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000
> --astf -c 1
>
>
>
> 1) with the issue (without the patch)
> *itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | *#define
> RTE_LIBRTE_I40E_ITR_INTERVAL -1
>
>
> latency : 40usec
>
> -Latency stats enabled
> Cpu Utilization : 0.2 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
>
> ----------------------------------------------------------------------
> -----------------------
> -------------------
> 0 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 41 41 34 40 41
> 33 43 8 8
> 1 | 5603, 5603, 0, 0, 12 , 44, 24 | 0 0 0 0 39 41 41 34 40
> 33 33 8 8
> 2 | 5603, 5603, 0, 0, 8 , 43, 5 | 0 0 0 0 38 41 42 40 40
> 40 42 9 8
> 3 | 5603, 5603, 0, 0, 6 , 43, 1 | 0 0 0 0 36 41 34 42 43
> 8 35 40 41
> *** TRex is shutting down - cause: 'CTRL + C detected'
>
> 2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4
>
> itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define
> RTE_LIBRTE_I40E_ITR_INTERVAL 4
>
> latency : 40usec
>
> -Latency stats enabled
> Cpu Utilization : 0.2 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
>
> ----------------------------------------------------------------------
> -----------------------
> -------------------
> 0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24
> 32 23 24 23
> 1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19
> 20 30 18 21
> 2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40
> 41 41 40 45
> 3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44
> 43 43 44 48
> *
>
>
> 3) RTE_LIBRTE_I40E_ITR_INTERVAL 0
>
> itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define
> RTE_LIBRTE_I40E_ITR_INTERVAL 0
>
> latency : 40usec
>
> -Latency stats enabled
> Cpu Utilization : 0.2 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
>
> ----------------------------------------------------------------------
> -----------------------
> -------------------
> 0 | 5034, 5034, 0, 0, 10 , 42, 0 | 0 0 0 0 0 42 40 38 24
> 32 23 24 23
> 1 | 5034, 5034, 0, 0, 8 , 43, 0 | 0 0 0 0 0 43 37 38 19
> 20 30 18 21
> 2 | 5034, 5034, 0, 0, 8 , 45, 0 | 0 0 0 0 0 37 40 41 40
> 41 41 40 45
> 3 | 5034, 5034, 0, 0, 8 , 48, 0 | 0 0 0 0 0 42 36 43 44
> 43 43 44 48
> *
>
> 4) TRex patch issue solved
>
> I40E_QINT_RQCTL_ITR_INDX_MASK
> #define RTE_LIBRTE_I40E_ITR_INTERVAL -1
>
> latency : 8usec
>
> -Latency stats enabled
> Cpu Utilization : 0.1 %
> if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter max
> window
> | , , , , average , max , (usec)
>
> ----------------------------------------------------------------------
> -----------------------
> -------------------
> 0 | 9501, 9501, 0, 0, 7 , 21, 0 | 21 9 9 12 9 9 14 12 8 9
> 9 9 8
> 1 | 9501, 9501, 0, 0, 7 , 25, 0 | 22 8 9 12 9 9 15 12 8 8
> 9 9 8
> 2 | 9501, 9501, 0, 0, 6 , 26, 0 | 22 9 10 9 10 10 15 13 8
> 9 9 9 8
> 3 | 9501, 9501, 0, 0, 6 , 32, 0 | 22 8 9 9 9 9 15 12 8 7
> 9 9 9
>
> Thanks,
> Hanoh
>
> From: Wu, Jingjing [mailto:jingjing.wu at intel.com]
> Sent: Tuesday, November 28, 2017 5:01 PM
> To: Hanoch Haim (hhaim)
> Cc: dev at dpdk.org<mailto:dev at dpdk.org>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi, Hanoch
>
> If DPDK PF, the commit affects that because it introduces an argument
> (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx
> with default value 32us.
>
> If you'd like to get the descriptor write back immediately, you can
> set "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file.
>
> Or you can just change the definition of
> I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
> #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
>
> Thanks
> Jingjing
>
> From: Hanoch Haim (hhaim) [mailto:hhaim at cisco.com]
> Sent: Tuesday, November 28, 2017 9:14 PM
> To: Wu, Jingjing <jingjing.wu at intel.com<mailto:jingjing.wu at intel.com>>
> Cc: dev at dpdk.org<mailto:dev at dpdk.org>; Hanoch Haim (hhaim)
> <hhaim at cisco.com<mailto:hhaim at cisco.com>>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi Jingjing,
>
> 1. The issue is with DPDK PF.
>
> 2. The rate is high ~10gb, one DP core, one latency core.
>
> 3. The fix is here
>
> /* Bind all RX queues to allocated MSIX interrupt */
> for (i = 0; i < nb_queue; i++) {
> val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
> #ifdef TREX_PATCH
>
> I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b = NoITR
> #else
> itr_idx <<
> I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies
> #endif
>
> I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
>
> The Interrupt Throttling ITR is configure using a different setting
> using a different register here :
>
>
> 4. The ITR_INTERVAL is 32 usec and it affect a different PF register
>
> #define I40E_ITR_INDEX_DEFAULT 0
> #define I40E_ITR_INDEX_NONE 3
> #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
>
>
> 5. My question is why the VF configuration affects PF INT_INTERVAL ? Can
> I remove my patch and fix this latency issue in the different way?
>
> Thanks,
> Hanoh
>
> From: Wu, Jingjing [mailto:jingjing.wu at intel.com]
> Sent: Tuesday, November 28, 2017 2:50 PM
> To: Hanoch Haim (hhaim); dev at dpdk.org<mailto:dev at dpdk.org>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi, Hanoch
>
> Are you talking about i40 VF's latency? And you are using DPDK PF as
> host driver?
>
> In this case, we are setting the Interrupt Throttling (ITR) to be maximum.
> That is to say, if the packet rate is very slow , the receive
> descriptor is written back when ITR timeout, otherwise it is written
> back when cache line is full (4 descriptors/packets). I think that's why you saw the latency is varying.
>
> If we change the ITR to minor, then huge number of interrupts will
> coming to core which impact performance.
>
>
> Thanks
> Jingjing
>
> From: Hanoch Haim (hhaim) [mailto:hhaim at cisco.com]
> Sent: Friday, November 24, 2017 7:25 PM
> To: dev at dpdk.org<mailto:dev at dpdk.org>
> Cc: Wu, Jingjing <jingjing.wu at intel.com<mailto:jingjing.wu at intel.com>>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Re-sending
>
> Hanoh
>
> From: Hanoch Haim (hhaim)
> Sent: Monday, November 20, 2017 5:19 PM
> To: dev at dpdk.org<mailto:dev at dpdk.org>
> Cc: Wu, Jingjing
> (jingjing.wu at intel.com<mailto:jingjing.wu at intel.com>);
> Hanoch Haim (hhaim)
> Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt
> throttling setting in PF
>
> Hi All,
> While integrating dpdk17.11 into TRex latest code a new latency issue
> is observed (i40e is very sensitive because it has very good
> resolution due to Qos configuration).
> git bitsec found the following commit.
> With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
> Any idea why?
> I can send how to reproduce this, it is very simple.
>
> cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
> commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
> Author: Jingjing Wu <jingjing.wu at intel.com<mailto:jingjing.wu at intel.com>>
> Date: Thu Aug 24 09:57:51 2017 +0800
>
> net/i40e: fix interrupt throttling setting in PF
>
> As no matter the PF host driver is DPDK or other kernel drivers,
> they are sharing the same virtchnnl interfaces to communicate to VFs.
> To follow the generic interface, DPDK PF need to set Interrupt
> Throttling (ITR) index according to the rxitr_idx from virtchnnl
> instead of ITR_NONE.
>
> Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support
> Linux
> VF")
> Cc: stable at dpdk.org<mailto:stable at dpdk.org>
>
> Signed-off-by: Jingjing Wu
> <jingjing.wu at intel.com<mailto:jingjing.wu at intel.com>>
>
>
>
> Thanks,
> Hanoh
More information about the dev
mailing list