<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:DengXian;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:DengXian;}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=ZH-CN style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Hi all,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> I was measuring the latency with the new Intel E810. I first used testpmd application with a single core and a single pair of queues and measured the latency on the generator side. The problem is that a latency bump occurs when the background traffic is above a certain threshold. I noticed that the threshold would move (at different rate of background traffic) depending on the speed of the recv and xmit function (i.e. bulk, SSE or AVX2)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> To identify where the bump occurs, I added hardware timestamp support to the application. I enabled rx hardware timestamp offload capability of E810, used <i>rte_eth_timesync_read_time </i>after <i>rte_eth_rx_burst </i>returns and <i>rte_eth_timesync_read_tx_timestamp </i>after <i>rte_eth_tx_burst </i>returns. I found the latency bump occurs between the packet arrives at the PHY Core and <i>rte_eth_tx_burst </i>returns. I also measures the CPU cycles before <i>rte_eth_rx_burst </i>is called and <i>rte_eth_tx_burst </i>returns in the user space. The gap in CPU cycles is stable regardless of the background traffic. This means the bump resides between the packet arrives the NIC and the packet is extracted from the main memory via <i>rte_eth_rx_burst.<o:p></o:p></i></span></p><p class=MsoNormal><span lang=EN-US> Meanwhile I failed to find any DPDK latency report from Intel nor mails from those who might experience the same problem. Does anyone meet the same problem and probably know what happens between the packet is in the PHY Core and the packet is in the memory? Maybe Intel Validation Team?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> I guess it may relate to packet discarding logic in the firmware or the DMA process. I saw this issue on different servers and different versions of firmware or DDP as well.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Configuration of the server:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>CPU: Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>RAM: DDR4 11 x 32 GB, 2933 MHz, 6 Channels<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>OS: Ubuntu 20.04.2 LTS<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Kernel: 5.4.0-89-generic<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Ice kernel driver version: 1.6.7<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>OS default DDP version: 1.3.26<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Firmware version: 3.0<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Traffic generator: MoonGen with two Mellanox ConnectX-5 EN 100G NICs<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US> Best Wishes<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> Pan<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div></body></html>