<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hi Michal,<o:p></o:p> <o:p> </o:p> I'll "top post" on this reply as the content is in HTML format below. In future, please try to send plain-text emails to DPDK mailing lists.<o:p></o:p> <o:p> </o:p> Regarding the issue you're having, its interesting that allocating from hugepage backed memory "solves" the problem, even when going<o:p></o:p> back to the lower traffic rate. The main difference for a CPU to access hugepage backed or 4k paged backed memory is the DTLB[1] pressure.<o:p></o:p> <o:p> </o:p> In your scenario, both page-sizes work equally well at the start (no drops). This is likely as all buffers are being accessed linearly,<o:p></o:p> and there are no packet drops, resulting in good re-use of buffers.<o:p></o:p> <o:p> </o:p> Lets discuss the 4K page scenario:<o:p></o:p> When the rate is turned up, packets are dropped, and the CPU(s) cannot keep up. This results in NIC rx descriptor rings being totally<o:p></o:p> full of used packets, and the mempools that contain the buffers become more "fragmented" in that not every buffer is on the same<o:p></o:p> 4k page anymore. In the worst case, each mbuf could be on a _different_ 4k page!<o:p></o:p> <o:p> </o:p> I think that when turning down the rate again, the fragmentation of mbufs in the mempool remains, resulting in continued loss of packets.<o:p></o:p> <o:p> </o:p> Estimating and talking is never conclusive – lets measure using Linux "Perf" tool. Run this command 3x, just like you posted the drop stats below.<o:p></o:p> I expect to see lower dTLB-load-misses on the first run (no drops, 10 mpps), and that the dTLB misses are higher for 15 mpps *and* for 10 mpps again afterwards.<o:p></o:p> perf stat -e cycles,dTLB-load-misses -C <datapath_lcore_here> -- sleep 1<o:p></o:p> <o:p> </o:p> Please try the commands, and report back your findings! Hope that helps, -Harry<o:p></o:p> <o:p> </o:p> [1] TLB & DPDK Resources;<o:p></o:p> <a href="https://en.wikipedia.org/wiki/Translation_lookaside_buffer">https://en.wikipedia.org/wiki/Translation_lookaside_buffer</a> (DTLB just means Data-TLB, as opposed to instruction-TLB)<o:p></o:p> <a href="https://stackoverflow.com/questions/52077230/huge-number-of-dtlb-load-misses-when-dpdk-forwarding-test">https://stackoverflow.com/questions/52077230/huge-number-of-dtlb-load-misses-when-dpdk-forwarding-test</a><o:p></o:p> <a href="https://www.dpdk.org/wp-content/uploads/sites/35/2018/12/LeiJiayu_Revise-4K-Pages-Performance-Impact-For-DPDK-Applications.pdf">https://www.dpdk.org/wp-content/uploads/sites/35/2018/12/LeiJiayu_Revise-4K-Pages-Performance-Impact-For-DPDK-Applications.pdf</a><o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Michał Niciejewski <michal.niciejewski@codilime.com> Sent: Wednesday, December 22, 2021 9:57 AM To: users@dpdk.org Subject: Unexpected behavior when using mbuf pool with external buffers<o:p></o:p> </div> </div> <o:p> </o:p> <div> Hi, recently I stumbled upon a problem with mbuf pool with external buffers. I allocated some memory with aligned_alloc(), registered it, DMA mapped the memory, and created mbuf pool: size_t mem_size = RTE_ALIGN_CEIL(MBUFS_NUM * QUEUE_NUM * RTE_MBUF_DEFAULT_BUF_SIZE, 4096); auto mem = aligned_alloc(4096, mem_size); mlock(mem, mem_size); rte_pktmbuf_extmem ext_mem = { .buf_ptr = mem, .buf_iova = (uintptr_t)mem, .buf_len = mem_size, .elt_size = RTE_MBUF_DEFAULT_BUF_SIZE, }; if (rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len, nullptr, 0, 4096) != 0) throw runtime_error("Failed to register DPDK external memory"); if (rte_dev_dma_map(dev, ext_mem.buf_ptr, ext_mem.buf_iova, ext_mem.buf_len) != 0) throw runtime_error("Failed to DMA map external memory"); mp = rte_pktmbuf_pool_create_extbuf("ext_mbuf_pool", MBUFS_NUM * QUEUE_NUM, 0, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_eth_dev_socket_id(0), &ext_mem, 1); if (mp == nullptr) throw runtime_error("Failed to create external mbuf pool"); The main loop of the program works like normal l2fwd: it receives packets and sends them to another port. std::vector<rte_mbuf *> mbufs(MAX_PKT_BURST); while (true) { auto rx_num = rte_eth_rx_burst(0, queue, mbufs.data(), MAX_PKT_BURST); if (!rx_num) continue; // ... auto tx_num = rte_eth_tx_burst(1, queue, mbufs.data(), rx_num); rte_pktmbuf_free_bulk(mbufs.data() + tx_num, rx_num - tx_num); } Every second, the program prints some info about the packets received in this second and some stats regarding rte_eth_tx_burst calls. For example, logs printed while receiving and sending 10mpps: Number of all rx burst calls: 12238365 Number of non-zero rx burst calls: 966834 Avg pkt nb received per rx burst: 0.816879 All received pkts: 9997264 All sent pkts: 9997264 All dropped pkts: 0 For lower traffic, everything looks fine. But when I start sending more packets some unexpected behavior occurs. When I increase traffic to 15mpps most of the packets are dropped on TX: Queue: 0 Number of rx burst calls: 4449541 Number of non-zero rx burst calls: 1616833 Avg pkt nb received per rx burst: 3.36962 All received pkts: 14993272 All sent pkts: 5827744 All dropped pkts: 9165528 After that, I checked again the results for 10mpps. Even though previously the application didn't have any troubles in managing 10mpps, now it does: Queue: 0 Number of all rx burst calls: 8722385 Number of non-zero rx burst calls: 1447741 Avg pkt nb received per rx burst: 1.14617 All received pkts: 9997316 All sent pkts: 8194416 All dropped pkts: 1802900 So basically it looks like sending too many packets breaks something and starts causing problems when sending fewer packets. I also tried allocating huge pages for mbuf pool instead of memory returned from aligned_alloc: auto mem = mmap(0, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0); And actually, it solved the problems - too big traffic doesn't affect lower traffic management. But I still want to know why memory allocated using aligned_alloc causes problems because in the place where I want to use mbuf pools with external buffers huge pages cannot be used like that. The full code used for testing: <a href="https://gist.github.com/tropuq/22625e0e5ac420a8ff5ae072a16f4c06" target="_blank">https://gist.github.com/tropuq/22625e0e5ac420a8ff5ae072a16f4c06</a> NIC used: Supermicro AOC-S25G-I2S-O Std Low Profile 25G Dual Port SFP28, based on Intel XXV710 Did anyone have similar issues or know what could cause such behavior? Is this allocation of the mbuf pool correct or am I missing something? Thanks in advance <o:p></o:p> <div> <o:p> </o:p> </div> -- <o:p></o:p> <div> <div> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="margin-left:10.75pt;border-collapse:collapse"> <tbody> <tr> <td style="padding:0cm 0cm 0cm 0cm"> Michał Niciejewski<o:p></o:p> </td> </tr> <tr> <td style="padding:0cm 0cm 0cm 0cm"> Junior Software Engineer<o:p></o:p> </td> </tr> <tr> <td style="padding:0cm 0cm 0cm 0cm"></td> </tr> </tbody> </table> </div> </div> </div> <div> <o:p> </o:p> </div> </div> </div> </body> </html>