<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style> </head> <body dir="ltr"> <p style="font-family:Arial;font-size:10pt;color:#008000;margin:15pt;font-style:normal;font-weight:normal;text-decoration:none;" align="Left"> [Public]<br> </p> <br> <div> <div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> Sharing a few critical points based on my exposure to the dma-perf application below</div> <div id="appendonsend"></div> <div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <span style="font-size: 11pt;"><br> </span></div> <div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <span style="font-size: 11pt;"><Snipped><br> <br> On Tue, Dec 12, 2023 at 04:16:20PM +0100, Morten Brørup wrote:<br> > +TO: Bruce, please stop me if I'm completely off track here.<br> ><br> > > From: Ferruh Yigit [<a href="mailto:ferruh.yigit@amd.com" id="OWAf90557d8-150f-cb2d-7de8-3c6a7c2889ad" class="OWAAutoLink" data-loopstyle="linkonly">mailto:ferruh.yigit@amd.com</a>] Sent: Tuesday, 12<br> > > December 2023 15.38<br> > ><br> > > On 12/12/2023 11:40 AM, Morten Brørup wrote:<br> > > >> From: Vipin Varghese [<a href="mailto:vipin.varghese@amd.com" id="OWA98c00610-3cd4-e752-037a-17ca12dfc14c" class="OWAAutoLink" data-loopstyle="linkonly">mailto:vipin.varghese@amd.com</a>] Sent: Tuesday,<br> > > >> 12 December 2023 11.38<br> > > >><br> > > >> Replace pktmbuf pool with mempool, this allows increase in MOPS<br> > > >> especially in lower buffer size. Using Mempool, allows to reduce the<br> > > >> extra CPU cycles.<br> > > ><br> > > > I get the point of this change: It tests the performance of copying<br> > > raw memory objects using respectively rte_memcpy and DMA, without the<br> > > mbuf indirection overhead.<br> > > ><br> > > > However, I still consider the existing test relevant: The performance<br> > > of copying packets using respectively rte_memcpy and DMA.<br> > > ><br> > ><br> > > This is DMA performance test application and packets are not used,<br> > > using pktmbuf just introduces overhead to the main focus of the<br> > > application.<br> > ><br> > > I am not sure if pktmuf selected intentionally for this test<br> > > application, but I assume it is there because of historical reasons.<br> ><br> > I think pktmbuf was selected intentionally, to provide more accurate<br> > results for application developers trying to determine when to use<br> > rte_memcpy and when to use DMA. Much like the "copy breakpoint" in Linux<br> > Ethernet drivers is used to determine which code path to take for each<br> > received packet.</span></div> <div class="elementToProof"><span style="font-size: 11pt;"><br> </span></div> <div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">yes Ferruh, this is the right understanding. In DPDK example we already have </span></div> <div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">dma-forward application which makes use of pktmbuf payload to copy over</span></div> <div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">new pktmbuf payload area. </span></div> <div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br> </span></div> <div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">by moving to mempool, we are actually now focusing on source and destination buffers.</span></div> <div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">This allows to create mempool objects with 2MB and 1GB src-dst areas. Thus allowing</span></div> <div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">to focus src to dst copy. With pktmbuf we were not able to achieve the same.</span></div> <div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><br> </span></div> <div class="elementToProof"><span style="font-size: 11pt;"><br> ><br> > Most applications will be working with pktmbufs, so these applications<br> > will also experience the pktmbuf overhead. Performance testing with the<br> > same overhead as the application will be better to help the application<br> > developer determine when to use rte_memcpy and when to use DMA when<br> > working with pktmbufs.</span></div> <div class="elementToProof"><span style="font-size: 11pt;"><br> </span></div> <div class="elementToProof"><span style="font-size: 11pt;">Morten thank you for the input, but as shared above DPDK example dma-fwd does </span></div> <div class="elementToProof"><span style="font-size: 11pt;">justice to such scenario. inline to test-compress-perf & test-crypto-perf IMHO test-dma-perf</span></div> <div class="elementToProof"><span style="font-size: 11pt;">should focus on getting best values of dma engine and memcpy comparision.</span></div> <div class="elementToProof"><span style="font-size: 11pt;"><br> ><br> > (Furthermore, for the pktmbuf tests, I wonder if copying performance<br> > could also depend on IOVA mode and RTE_IOVA_IN_MBUF.)<br> ><br> > Nonetheless, there may also be use cases where raw mempool objects are<br> > being copied by rte_memcpy or DMA, so adding tests for these use cases<br> > are useful.<br> ><br> ><br> > @Bruce, you were also deeply involved in the DMA library, and probably<br> > have more up-to-date practical experience with it. Am I right that<br> > pktmbuf overhead in these tests provides more "real life use"-like<br> > results? Or am I completely off track with my thinking here, i.e. the<br> > pktmbuf overhead is only noise?<br> ><br> I'm actually not that familiar with the dma-test application, so can't<br> comment on the specific overhead involved here. In the general case, if we<br> are just talking about the overhead of dereferencing the mbufs then I would<br> expect the overhead to be negligible. However, if we are looking to include<br> the cost of allocation and freeing of buffers, I'd try to avoid that as it<br> is a cost that would have to be paid for both SW copies and HW copies, so<br> should not count when calculating offload cost.</span></div> <div class="elementToProof"><span style="font-size: 11pt;"><br> </span></div> <div class="elementToProof"><span style="font-size: 11pt;">Bruce, as per test-dma-perf there is no repeated pktmbuf-alloc or pktmbuf-free. </span></div> <div class="elementToProof"><span style="font-size: 11pt;">Hence I disagree that the overhead discussed for pkmbuf here is not related to alloc and free.</span></div> <div class="elementToProof"><span style="font-size: 11pt;">But the cost as per my investigation goes into fetching the cacheline and performing mtod on</span></div> <div class="elementToProof"><span style="font-size: 11pt;">each iteration.<br> <br> /Bruce<br> <br> </span></div> <div class="elementToProof"><span style="font-size: 11pt;">I can rewrite the logic to make use pktmbuf objects by sending the src and dst with pre-computed </span></div> <div class="elementToProof"><span style="font-size: 11pt;">mtod to avoid the overhead. But this will not resolve the 2MB and 1GB huge page copy alloc failures.</span></div> <div class="elementToProof"><span style="font-size: 11pt;">IMHO, I believe in similar lines to other perf application, dma-perf application should focus on acutal device</span></div> <div class="elementToProof"><span style="font-size: 11pt;">performance over application application performance.</span></div> </div> </body> </html>