<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word"> <div class="WordSection1"> Hi,<o:p></o:p> I have run out of ideas and thought I would reach out to the dpdk community.<o:p></o:p> <o:p> </o:p> I have a Sapphire Rapids dual CPU server and one E180 (also tried X710), both are 4x10G NICs. When our application pipeline final stage enqueues mbufs into the tx ring I expect the rte_ring_dequeue_burst() to pull the mbufs from the tx ring and rte_eth_tx_burst() transmit them at line rate. What I see is when there is one interface receiving 64-byte UDP in IPv4 the receive and transmit is at line rate (i.e. packets in one port and out another port of the NIC @14.9 MPPS).<o:p></o:p> When I turn on another receive port then both transmit ports of the NIC shows Tx performance drops to 5 MPPS. The Tx ring is filling faster than Tx thread can dequeue and transmit mbufs. <o:p></o:p> <o:p> </o:p> Packets arrive on ports 1 and 3 in my test setup. NIC is on NUMA Node 1. Hugepage memory (6GB, 1GB page size) is on NUMA Node 1. The mbuf size is 9KB.<o:p></o:p> <o:p> </o:p> Rx Port 1 -> Tx Port 2<o:p></o:p> Rx Port 3 -> Tx port 4<o:p></o:p> <o:p> </o:p> I monitor the mbufs available and they are:<o:p></o:p> *** DPDK Mempool Configuration ***<o:p></o:p> Number Sockets : 1<o:p></o:p> Memory/Socket GB : 6<o:p></o:p> Hugepage Size MB : 1024<o:p></o:p> Overhead/socket MB : 512<o:p></o:p> Usable mem/socket MB: 5629<o:p></o:p> mbuf size Bytes : 9216<o:p></o:p> nb mbufs per socket : 640455<o:p></o:p> total nb mbufs : 640455<o:p></o:p> hugepages/socket GB : 6<o:p></o:p> mempool cache size : 512<o:p></o:p> <o:p> </o:p> *** DPDK EAL args ***<o:p></o:p> EAL lcore arg : -l 36 <<< NUMA Node 1<o:p></o:p> EAL socket-mem arg : --socket-mem=0,6144<o:p></o:p> <o:p> </o:p> The number of rings in this configuration is 16 and all are the same size (16384 * 8), and there is one mempool.<o:p></o:p> <o:p> </o:p> The Tx rings are created as SP and SC when created.<o:p></o:p> <o:p> </o:p> There is one Tx thread per NIC port, where its only task is to dequeue mbufs from the tx ring and call rte_eth_tx_burst() to transmit the mbufs. The dequeue burst size is 512 and tx burst is equal to or less than 512. The rte_eth_tx_burst() never returns less than the bust size given.<o:p></o:p> <o:p> </o:p> Each Tx thread is on a dedicated CPU core and its sibling is unused.<o:p></o:p> We use cpushielding to keep noncritical threads from using these CPUs for Tx threads. HTOP shows the Tx threads are the only threads using the carved-out CPUs.<o:p></o:p> <o:p> </o:p> In the Tx thread it uses the rte_ring_dequeue_burst() to get a burst of mbufs up to 512. <o:p></o:p> I added debug counters to keep track of how many mbufs are dequeued from the tx ring with rte_ring_dequeue_burst() that equals to the 512 and a counter for less than 512. The dequeue of the tx ring is always 512, never less. <o:p></o:p> <o:p> </o:p> <o:p> </o:p> Note: if I skip the rte_eth_tx_burst() in the Tx threads and just dequeue the mbufs and bulk free the mbufs from the tx ring I do not see the tx ring fill-up, i.e., it is able to free the mbufs faster than they arrive on the tx ring.<o:p></o:p> <o:p> </o:p> So, I suspect that the rte_eth_tx_burst() is the bottleneck to investigate, which involves the inner bows of DPDK and Intel NIC architecture.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> Any help to resolve my issue is greatly appreciated.<o:p></o:p> <o:p> </o:p> Thanks,<o:p></o:p> Ed<o:p></o:p> <o:p> </o:p> <o:p></o:p> <o:p> </o:p> <o:p> </o:p> </div> </body> </html>