<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hi,<o:p></o:p> When I return to 2K mbuf size the /proc/meminfo hugepage info looks exactly the same.<o:p></o:p> HugePages_Total: 1024<o:p></o:p> HugePages_Free: 0<o:p></o:p> HugePages_Rsvd: 0<o:p></o:p> HugePages_Surp: 0<o:p></o:p> Hugepagesize: 2048 kB<o:p></o:p> <o:p> </o:p> I did not make any changes to the arguments to rte_eal_init() when I tried 16K mbuf configuration.<o:p></o:p> <o:p> </o:p> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: Cliff Burdick <shaklee3@gmail.com> Sent: Tuesday, March 1, 2022 10:57 PM To: Lombardo, Ed <Ed.Lombardo@netscout.com> Cc: Stephen Hemminger <stephen@networkplumber.org>; users@dpdk.org Subject: Re: How to increase mbuf size in dpdk version 17.11<o:p></o:p> </div> <o:p> </o:p> <div> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#006068"> <tbody> <tr> <td style="padding:3.75pt 3.75pt 3.75pt 3.75pt"> External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p> </td> </tr> </tbody> </table> </div> <div> That's showing you have 0 hugepages free. Maybe they weren't passed through to the VM properly?<o:p></o:p> </div> <o:p> </o:p> <div> <div> On Tue, Mar 1, 2022 at 7:50 PM Lombardo, Ed <<a href="mailto:Ed.Lombardo@netscout.com">Ed.Lombardo@netscout.com</a>> wrote:<o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> <div> <div> [root@vSTREAM_632 ~]# cat /proc/meminfo<o:p></o:p> MemTotal: 32778372 kB<o:p></o:p> MemFree: 15724124 kB<o:p></o:p> MemAvailable: 15897392 kB<o:p></o:p> Buffers: 18384 kB<o:p></o:p> Cached: 526768 kB<o:p></o:p> SwapCached: 0 kB<o:p></o:p> Active: 355140 kB<o:p></o:p> Inactive: 173360 kB<o:p></o:p> Active(anon): 62472 kB<o:p></o:p> Inactive(anon): 12484 kB<o:p></o:p> Active(file): 292668 kB<o:p></o:p> Inactive(file): 160876 kB<o:p></o:p> Unevictable: 13998696 kB<o:p></o:p> Mlocked: 13998696 kB<o:p></o:p> SwapTotal: 3906556 kB<o:p></o:p> SwapFree: 3906556 kB<o:p></o:p> Dirty: 76 kB<o:p></o:p> Writeback: 0 kB<o:p></o:p> AnonPages: 13986156 kB<o:p></o:p> Mapped: 95500 kB<o:p></o:p> Shmem: 16864 kB<o:p></o:p> Slab: 121952 kB<o:p></o:p> SReclaimable: 71128 kB<o:p></o:p> SUnreclaim: 50824 kB<o:p></o:p> KernelStack: 4608 kB<o:p></o:p> PageTables: 31524 kB<o:p></o:p> NFS_Unstable: 0 kB<o:p></o:p> Bounce: 0 kB<o:p></o:p> WritebackTmp: 0 kB<o:p></o:p> CommitLimit: 19247164 kB<o:p></o:p> Committed_AS: 14170424 kB<o:p></o:p> VmallocTotal: 34359738367 kB<o:p></o:p> VmallocUsed: 212012 kB<o:p></o:p> VmallocChunk: 34342301692 kB<o:p></o:p> Percpu: 2816 kB<o:p></o:p> HardwareCorrupted: 0 kB<o:p></o:p> AnonHugePages: 13228032 kB<o:p></o:p> CmaTotal: 0 kB<o:p></o:p> CmaFree: 0 kB<o:p></o:p> HugePages_Total: 1024<o:p></o:p> HugePages_Free: 0<o:p></o:p> HugePages_Rsvd: 0<o:p></o:p> HugePages_Surp: 0<o:p></o:p> Hugepagesize: 2048 kB<o:p></o:p> DirectMap4k: 104320 kB<o:p></o:p> DirectMap2M: 33449984 kB<o:p></o:p> <o:p></o:p> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: Cliff Burdick <<a href="mailto:shaklee3@gmail.com" target="_blank">shaklee3@gmail.com</a>> Sent: Tuesday, March 1, 2022 10:45 PM To: Lombardo, Ed <<a href="mailto:Ed.Lombardo@netscout.com" target="_blank">Ed.Lombardo@netscout.com</a>> Cc: Stephen Hemminger <<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>>; <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> Subject: Re: How to increase mbuf size in dpdk version 17.11<o:p></o:p> </div> <o:p></o:p> <div> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#006068"> <tbody> <tr> <td style="padding:3.75pt 3.75pt 3.75pt 3.75pt"> External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p> </td> </tr> </tbody> </table> </div> <div> Can you paste the output of "cat /proc/meminfo"?<o:p></o:p> </div> <o:p></o:p> <div> <div> On Tue, Mar 1, 2022 at 5:37 PM Lombardo, Ed <<a href="mailto:Ed.Lombardo@netscout.com" target="_blank">Ed.Lombardo@netscout.com</a>> wrote:<o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> Here is the output from rte_mempool_dump() after creating the mbuf " mbuf_pool_create (mbuf_seg_size=16512, nb_mbuf=32768, socket_id=0)": nb_mbuf_per_pool = 32768 mb_size = 16640 16512 * 32768 = 541,065,216 mempool <mbuf_pool_socket_0>@0x17f811400 flags=10 pool=0x17f791180 iova=0x80fe11400 nb_mem_chunks=1 size=32768 populated_size=32768 header_size=64 elt_size=16640 trailer_size=0 total_obj_size=16704 private_data_size=64 avg bytes/object=16704.000000 internal cache infos: cache_size=250 cache_count[0]=0 ... cache_count[126]=0 cache_count[127]=0 total_cache_count=0 common_pool_count=32768 no statistics available -----Original Message----- From: Stephen Hemminger <<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>> Sent: Tuesday, March 1, 2022 5:46 PM To: Cliff Burdick <<a href="mailto:shaklee3@gmail.com" target="_blank">shaklee3@gmail.com</a>> Cc: Lombardo, Ed <<a href="mailto:Ed.Lombardo@netscout.com" target="_blank">Ed.Lombardo@netscout.com</a>>; <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> Subject: Re: How to increase mbuf size in dpdk version 17.11 External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Tue, 1 Mar 2022 13:37:07 -0800 Cliff Burdick <<a href="mailto:shaklee3@gmail.com" target="_blank">shaklee3@gmail.com</a>> wrote: > Can you verify how many buffers you're allocating? I don't see how > many you're allocating in this thread. > > On Tue, Mar 1, 2022 at 1:30 PM Lombardo, Ed <<a href="mailto:Ed.Lombardo@netscout.com" target="_blank">Ed.Lombardo@netscout.com</a>> > wrote: > > > Hi Stephen, > > The VM is configured to have 32 GB of memory. > > Will dpdk consume the 2GB of hugepage memory for the mbufs? > > I don't mind having less mbufs with mbuf size of 16K vs original > > mbuf size of 2K. > > > > Thanks, > > Ed > > > > -----Original Message----- > > From: Stephen Hemminger <<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>> > > Sent: Tuesday, March 1, 2022 2:57 PM > > To: Lombardo, Ed <<a href="mailto:Ed.Lombardo@netscout.com" target="_blank">Ed.Lombardo@netscout.com</a>> > > Cc: <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> > > Subject: Re: How to increase mbuf size in dpdk version 17.11 > > > > External Email: This message originated outside of NETSCOUT. Do not > > click links or open attachments unless you recognize the sender and > > know the content is safe. > > > > On Tue, 1 Mar 2022 18:34:22 +0000 > > "Lombardo, Ed" <<a href="mailto:Ed.Lombardo@netscout.com" target="_blank">Ed.Lombardo@netscout.com</a>> wrote: > > > > > Hi, > > > I have an application built with dpdk 17.11. > > > During initialization I want to change the mbuf size from 2K to 16K. > > > I want to receive packet sizes of 8K or more in one mbuf. > > > > > > The VM running the application is configured to have 2G hugepages. > > > > > > I tried many things and I get an error when a packet arrives. > > > > > > I read online that there is #define DEFAULT_MBUF_DATA_SIZE that I > > changed from 2176 to ((2048*8)+128), where 128 is for headroom. > > > The call to rte_pktmbuf_pool_create() returns success with my changes. > > > From the rte_mempool_dump() - "rx_nombuf" - Total number of Rx > > > mbuf > > allocation failures. This value increments each time a packet arrives. > > > > > > Is there any reference document explaining what causes this error? > > > Is there a user guide I should follow to make the mbuf size > > > change, > > starting with the hugepage value? > > > > > > Thanks, > > > Ed > > > > Did you check that you have enough memory in the system for the > > larger footprint? > > Using 16K per mbuf is going to cause lots of memory to be consumed. A little maths you can fill in your own values. Assuming you want 16K of data. You need at a minimum [1] num_rxq := total number of receive queues num_rxd := number of receive descriptors per receive queue num_txq := total number of transmit queues (assume all can be full) num_txd := number of transmit descriptors num_mbufs = num_rxq * num_rxd + num_txq * num_txd + num_cores * burst_size Assuming you are using code copy/pasted from some example like l3fwd. With 4 Rxq num_mbufs = 4 * 1024 + 4 * 1024 + 4 * 32 = 8320 Each mbuf element requires [2] elt_size = sizeof(struct rte_mbuf) + HEADROOM + mbuf_size = 128 + 128 + 16K = 16640 obj_size = rte_mempool_calc_obj_size(elt_size, 0, NULL) = 16832 So total pool is num_mbufs * obj_size = 8320 * 16832 = 140,042,240 ~ 139M [1] Some devices line bnxt need multiple buffers per packet. [2] Often applications want additional space per mbuf for meta-data. <o:p></o:p> </blockquote> </div> </div> </div> </blockquote> </div> </div> </body> </html>