<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="KO" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> <div> Hi,<o:p></o:p> <o:p> </o:p> Packets seem to be silently discarded in the RX NIC while waiting in the buffer, and it does not happen when I use only one worker core.<o:p></o:p> I think multiple worker cores packet processing would have this problem to handle heavy traffic in parallel.<o:p></o:p> <o:p> </o:p> I want to increase buffer size to solve this as below, then is it right to change only modify the number from 4 to 8?<o:p></o:p> I am trying to gather DPDK and PMD driver log such as "--log-level="7" --log-level="pmd.net,7". Could you recommend how to set the log-level string to confirm silent packet drop?<o:p></o:p> <o:p> </o:p> dpdk-stable-20.11.1/lib/librte_eventdev/rte_event_eth_rx_adapter.c<o:p></o:p> <o:p> </o:p> #define BATCH_SIZE 32<o:p></o:p> #define BLOCK_CNT_THRESHOLD 10<o:p></o:p> #define ETH_EVENT_BUFFER_SIZE (4*BATCH_SIZE) // e.g. (8*BATCH_SIZE)<o:p></o:p> <o:p> </o:p> #define ETH_RX_ADAPTER_SERVICE_NAME_LEN 32<o:p></o:p> #define ETH_RX_ADAPTER_MEM_NAME_LEN 32<o:p></o:p> <o:p> </o:p> #define RSS_KEY_SIZE 40<o:p></o:p> /* value written to intr thread pipe to signal thread exit */<o:p></o:p> #define ETH_BRIDGE_INTR_THREAD_EXIT 1<o:p></o:p> /* Sentinel value to detect initialized file handle */<o:p></o:p> #define INIT_FD -1<o:p></o:p> <o:p> </o:p> BR/Jaeeun<o:p></o:p> </div> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Jaeeun Ham Sent: Wednesday, June 22, 2022 11:14 AM To: Jayatheerthan, Jay <jay.jayatheerthan@intel.com>; dev@dpdk.org Cc: Jerin Jacob <jerinj@marvell.com> Subject: RE: ask for TXA_FLUSH_THRESHOLD change<o:p></o:p> </div> </div> <o:p> </o:p> <div> Hi,<o:p></o:p> <o:p> </o:p> Could you guide me on how to eliminate or reduce tx drop/retry?<o:p></o:p> TXA_FLUSH_THRESHOLD helped somewhat but it was not cleared packet tx drop.<o:p></o:p> <o:p> </o:p> ========[ TX adapter stats ]========<o:p></o:p> tx_retry: 17499893<o:p></o:p> tx_packets: 7501716<o:p></o:p> tx_dropped: 5132458<o:p></o:p> <o:p> </o:p> BR/Jaeeun<o:p></o:p> </div> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Jayatheerthan, Jay <<a href="mailto:jay.jayatheerthan@intel.com">jay.jayatheerthan@intel.com</a>> Sent: Thursday, June 16, 2022 3:40 PM To: Jaeeun Ham <<a href="mailto:jaeeun.ham@ericsson.com">jaeeun.ham@ericsson.com</a>>; <a href="mailto:dev@dpdk.org">dev@dpdk.org</a> Cc: Jerin Jacob <<a href="mailto:jerinj@marvell.com">jerinj@marvell.com</a>> Subject: RE: ask for TXA_FLUSH_THRESHOLD change<o:p></o:p> </div> </div> <o:p> </o:p> Hi Jaeeun,<o:p></o:p> See my responses inline below.<o:p></o:p> <o:p> </o:p> -Jay<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Jaeeun Ham <<a href="mailto:jaeeun.ham@ericsson.com">jaeeun.ham@ericsson.com</a>> Sent: Monday, June 13, 2022 5:51 AM To: <a href="mailto:dev@dpdk.org">dev@dpdk.org</a> Cc: Jerin Jacob <<a href="mailto:jerinj@marvell.com">jerinj@marvell.com</a>>; Jayatheerthan, Jay <<a href="mailto:jay.jayatheerthan@intel.com">jay.jayatheerthan@intel.com</a>> Subject: ask for TXA_FLUSH_THRESHOLD change<o:p></o:p> </div> </div> <o:p> </o:p> Hi,<o:p></o:p> <o:p> </o:p> There were latency delay when I increase dpdk(20.11.1) core. (one worker core was okay.)<o:p></o:p> When I decrease the TXA_FLUSH_THRESHOLD value(1024 to 32), it was okay <o:p></o:p> <o:p> </o:p> It’s TXA_FLUSH_THRESHOLD at lib/librte_eventdev/rte_event_eth_tx_adapter.c. // <a href="https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-dff86f66a37f4e0e&q=1&e=9d9b2326-a7ae-405b-bc88-9a28d777c306&u=https%3A%2F%2Fgit.dpdk.org%2Fdpdk-stable%2Ftree%2Flib%2Flibrte_eventdev%2Frte_event_eth_tx_adapter.c%3Fh%3D20.11%23n15"> https://git.dpdk.org/dpdk-stable/tree/lib/librte_eventdev/rte_event_eth_tx_adapter.c?h=20.11#n15</a><o:p></o:p> When TXA_FLUSH_THRESHOLD value was changed from 1024 to 32, the latency test result was fine on 10 cores for low traffic(DL:20Mbps/UL:17kbps).<o:p></o:p> I think that this can make call for rte_eth_tx_buffer_flush() more frequently.<o:p></o:p> But, I’m not sure whether this approach can cause worse performance or not.<o:p></o:p> Do you have any opinion about this?<o:p></o:p> <o:p> </o:p> [Jay] Yes, it will cause rte_eth_tx_buffer_flush() to be called more often. It can lead to lesser batching benefit. Typical performance vs. latency trade-off decision apply here.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Similar RDK RTE_BRIDGE_ETH_TX_FLUSH_THOLD is patched on DUSG3 from 1024 to smaller value since DPDK 18.11.2:<o:p></o:p> I’m not aware of any side-effect, I think it is needed to have low enough latency even at low traffic rates. For more details see Intel <a href="https://footprints.intel.com/MRcgi/MRlogin.pl?DL=22288DA14">FP 22288</a>.<o:p></o:p> <o:p> </o:p> [Jay] Currently, TXA_FLUSH_THRESHOLD is not a configurable attr.<o:p></o:p> <o:p> </o:p> TXA_MAX_NB_TX(128) looks the same as CONFIG_RTE_BRIDGE_ETH_EVENT_BUFFER_SIZE(16384), then is it also should be tuned?<o:p></o:p> <o:p> </o:p> [Jay] They both are different attributes. TXA_MAX_NB_TX refers to max number of queues in Tx adapter. CONFIG_RTE_BRIDGE_ETH_EVENT_BUFFER_SIZE refers to event buffer size in Rx BD.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> --- dpdk-3pp-swu-18.11/dpdk-stable-18.11.2/config/common_base.orig 2020-01-29 15:05:10.000000000 +0100<o:p></o:p> +++ dpdk-3pp-swu-18.11/dpdk-stable-18.11.2/config/common_base 2020-01-29 15:11:10.000000000 +0100<o:p></o:p> @@ -566,9 +566,9 @@<o:p></o:p> CONFIG_RTE_LIBRTE_BRIDGE_ETH_MAX_CP_ENQ_RETRIES=100<o:p></o:p> CONFIG_RTE_MAX_BRIDGE_ETH_INSTANCE=4<o:p></o:p> CONFIG_RTE_BRIDGE_ETH_INTR_RING_SIZE=32<o:p></o:p> -CONFIG_RTE_BRIDGE_ETH_EVENT_BUFFER_SIZE=128<o:p></o:p> +CONFIG_RTE_BRIDGE_ETH_EVENT_BUFFER_SIZE=16384<o:p></o:p> CONFIG_RTE_LIBRTE_BRIDGE_ETH_DEBUG=n<o:p></o:p> -CONFIG_RTE_BRIDGE_ETH_TX_FLUSH_THOLD=1024<o:p></o:p> +CONFIG_RTE_BRIDGE_ETH_TX_FLUSH_THOLD=32<o:p></o:p> <o:p> </o:p> --- dpdk-3pp-swu-dusg3-20.11.3/dpdk-stable-20.11.3/config/rte_config.h 2021-08-05 23:46:52.051051000 +0200<o:p></o:p> +++ dpdk-3pp-swu-dusg3-20.11.3/dpdk-stable-20.11.3/config/rte_config.h 2021-08-06 00:50:07.310766255 +0200<o:p></o:p> @@ -175,8 +175,8 @@<o:p></o:p> #define RTE_LIBRTE_BRIDGE_ETH_MAX_CP_ENQ_RETRIES 100<o:p></o:p> #define RTE_MAX_BRIDGE_ETH_INSTANCE 4<o:p></o:p> #define RTE_BRIDGE_ETH_INTR_RING_SIZE 32<o:p></o:p> -#define RTE_BRIDGE_ETH_EVENT_BUFFER_SIZE 128<o:p></o:p> -#define RTE_BRIDGE_ETH_TX_FLUSH_THOLD 1024<o:p></o:p> +#define RTE_BRIDGE_ETH_EVENT_BUFFER_SIZE 16384<o:p></o:p> +#define RTE_BRIDGE_ETH_TX_FLUSH_THOLD 10<o:p></o:p> #undef RTE_BRIDGE_ETH_TX_MULTI_PKT_EVENT<o:p></o:p> <o:p> </o:p> <o:p> </o:p> BR/Jaeeun<o:p></o:p> <o:p> </o:p> </div> </div> </body> </html>