<div dir="ltr">Hello,<br>This is an update to this bug research, as I have had time to look at it again<div>I have created an example program (code below) and tried them with debug & rte_malloc_debug using dpdk 20.11 and 22.07<div>the results are the same - and will also be below</div><div>I now suspect it could be a bug in DPDK dynamic memory mode (it doesn't happen in legacy mode)</div><div>and may be related to a long allocation time causing a timeout</div><div>The application code is very minimal, and should at the most get an error at `rte_mempool_create`</div><div><br></div><div>more information about the system, firmware and DPDK compilation can be provided if it may be related to that</div><div><br>The primary process code:<br><div style="float:left;white-space:pre;line-height:1;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><span class="gmail-sc9" style="color:rgb(128,64,0);font-family:"Courier New";font-size:10pt">#include <rte_eal.h> #include <memory> </span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">int</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">main</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">void</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">)</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">{</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">const</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">char</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">*</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">flags</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">[]</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">=</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">{</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"-l"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"1"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"--no-pci"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">};</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">rte_eal_init</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">sizeof</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">flags</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">)</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">/</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">sizeof</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">char</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">*),</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">std</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">::</span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">const_cast</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt"><</span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">char</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">**>(</span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">flags</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">));</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">printf</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"primary started"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">);</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">while</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">true</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">)</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">{}</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">return</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc4" style="color:rgb(255,128,0);font-family:"Courier New";font-size:10pt">0</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">;</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">}</span></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>The secondary process code:</div><div><div style="float:left;white-space:pre;line-height:1;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><span class="gmail-sc9" style="color:rgb(128,64,0);font-family:"Courier New";font-size:10pt">#include <rte_eal.h> #include <rte_mempool.h> #include <memory> </span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">int</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">main</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">void</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">)</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">{</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">const</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">char</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">*</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">flags</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">[]</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">=</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">{</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"-l"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"1"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"--no-pci"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"--proc-type"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"secondary"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">};</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">rte_eal_init</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">sizeof</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">flags</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">)</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">/</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">sizeof</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">char</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">*),</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">std</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">::</span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">const_cast</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt"><</span><span class="gmail-sc16" style="color:rgb(128,0,255);font-family:"Courier New";font-size:10pt">char</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">**>(</span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">flags</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">));</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">rte_mempool</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">*</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">pool</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">=</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">rte_mempool_create</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"my_pool"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc4" style="color:rgb(255,128,0);font-family:"Courier New";font-size:10pt">150000000</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc4" style="color:rgb(255,128,0);font-family:"Courier New";font-size:10pt">40</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc4" style="color:rgb(255,128,0);font-family:"Courier New";font-size:10pt">0</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc4" style="color:rgb(255,128,0);font-family:"Courier New";font-size:10pt">0</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">NULL</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">NULL</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">NULL</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">NULL</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc4" style="color:rgb(255,128,0);font-family:"Courier New";font-size:10pt">0</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">,</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc4" style="color:rgb(255,128,0);font-family:"Courier New";font-size:10pt">0</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">); // 150M elements * 40B = 6GB mempool</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">if</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">pool</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">)</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">{</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">printf</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"allocation success"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">);</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">}</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">else</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">{</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">printf</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc6" style="color:rgb(128,128,128);font-family:"Courier New";font-size:10pt">"allocation failure"</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">);</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">}</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">fflush</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">(</span><span class="gmail-sc11" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)">stdout</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">);</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc5" style="font-weight:bold;color:rgb(0,0,255);font-family:"Courier New";font-size:10pt">return</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc4" style="color:rgb(255,128,0);font-family:"Courier New";font-size:10pt">0</span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">;</span><span class="gmail-sc0" style="font-family:"Courier New";font-size:10pt;color:rgb(0,0,0)"> </span><span class="gmail-sc10" style="font-weight:bold;color:rgb(0,0,128);font-family:"Courier New";font-size:10pt">}</span></div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>The result in the primary process:<br>EAL: Detected CPU lcores: 96</div></div></div><div>EAL: Detected NUMA nodes: 2</div><div>EAL: Detected shared linkage of DPDK</div><div>EAL: Multi-Process socket /var/run/dpdk/rte/mp_socket</div><div>EAL: Selected IOVA mode 'PA'</div><div>TELEMETRY: No legacy callbacks, legacy socket not created</div><div>primary started</div><div><br></div><div>The results in the secondary process:</div><div><div><div><div>EAL: Detected CPU lcores: 96</div></div></div><div>EAL: Detected NUMA nodes: 2</div><div>EAL: Detected shared linkage of DPDK</div><div>EAL: Multi-Process socket /var/run/dpdk/rte/mp_socket_.......</div><div>EAL: Selected IOVA mode 'PA'</div></div><div>EAL: Request timed out // <---------------This is the rte_mempool_create</div><div>EAL: Request timed out<br></div><div>EAL: Request timed out<br></div><div>*** crashes with retcode 139</div><div><br></div><div>The main process looks find from the CLI, but the secondary will not be able to start again (stuck at EAL: Selected IOVA mode 'PA')</div><div><br></div><div>What should my next step be? As far as debugging / solving / reporting this?</div></div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 31, 2022 at 2:32 PM MOD <<a href="mailto:sdk.register@gmail.com">sdk.register@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div>The issue is probably not with my code but with the compilation on DPDK,</div><div>because I got it to repeat on a separated program,</div><div>where I setup an EAL with the flags `-l 1 --no-pci`</div><div>(just rte_eal_init and rte_mempool_create)</div><div><br></div><div>this seems to be a memseg_list issue</div><div>When running the program above, and requesting large amounts of memory (200M elements of 8 bytes each)</div><div>I don't crash, but get `couldnt find suitable memseg_list` error</div><div>This also happens when trying to allocate from the main process</div><div><br></div><div>This error is probably related to these parameters from rte_config.h:<br><table style="border-spacing:0px;border-collapse:collapse;color:rgb(36,41,47);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-size:14px"><tbody style="box-sizing:border-box"><tr style="box-sizing:border-box"></tr><tr style="box-sizing:border-box;background-color:transparent"><td id="m_3847235289724525773gmail-LC30" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;white-space:pre-wrap"><span style="box-sizing:border-box"><span style="box-sizing:border-box">/*</span> EAL defines <span style="box-sizing:border-box">*/</span></span></td></tr><tr style="box-sizing:border-box"><td id="m_3847235289724525773gmail-L31" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;line-height:20px;text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_3847235289724525773gmail-LC31" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;white-space:pre-wrap">#<span style="box-sizing:border-box">define</span> <span style="box-sizing:border-box">RTE_MAX_HEAPS</span> <span style="box-sizing:border-box">32</span></td></tr><tr style="box-sizing:border-box;background-color:transparent"><td id="m_3847235289724525773gmail-L32" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;line-height:20px;text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_3847235289724525773gmail-LC32" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;white-space:pre-wrap">#<span style="box-sizing:border-box">define</span> <span style="box-sizing:border-box">RTE_MAX_MEMSEG_LISTS</span> <span style="box-sizing:border-box">128</span></td></tr><tr style="box-sizing:border-box"><td id="m_3847235289724525773gmail-L33" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;line-height:20px;text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_3847235289724525773gmail-LC33" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;white-space:pre-wrap">#<span style="box-sizing:border-box">define</span> <span style="box-sizing:border-box">RTE_MAX_MEMSEG_PER_LIST</span> <span style="box-sizing:border-box">8192</span></td></tr><tr style="box-sizing:border-box;background-color:transparent"><td id="m_3847235289724525773gmail-L34" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;line-height:20px;text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_3847235289724525773gmail-LC34" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;white-space:pre-wrap">#<span style="box-sizing:border-box">define</span> <span style="box-sizing:border-box">RTE_MAX_MEM_MB_PER_LIST</span> <span style="box-sizing:border-box">32768</span></td></tr><tr style="box-sizing:border-box"><td id="m_3847235289724525773gmail-L35" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;line-height:20px;text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_3847235289724525773gmail-LC35" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;white-space:pre-wrap">#<span style="box-sizing:border-box">define</span> <span style="box-sizing:border-box">RTE_MAX_MEMSEG_PER_TYPE</span> <span style="box-sizing:border-box">32768</span></td></tr><tr style="box-sizing:border-box;background-color:transparent"><td id="m_3847235289724525773gmail-L36" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;line-height:20px;text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_3847235289724525773gmail-LC36" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;white-space:pre-wrap">#<span style="box-sizing:border-box">define</span> <span style="box-sizing:border-box">RTE_MAX_MEM_MB_PER_TYPE</span> <span style="box-sizing:border-box">65536</span></td></tr><tr style="box-sizing:border-box"><td id="m_3847235289724525773gmail-L37" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;line-height:20px;text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_3847235289724525773gmail-LC37" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;white-space:pre-wrap">#<span style="box-sizing:border-box">define</span> <span style="box-sizing:border-box">RTE_MAX_MEMZONE</span> <span style="box-sizing:border-box">2560</span></td></tr><tr style="box-sizing:border-box;background-color:transparent"><td id="m_3847235289724525773gmail-L38" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;line-height:20px;text-align:right;white-space:nowrap;vertical-align:top"></td><td id="m_3847235289724525773gmail-LC38" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;overflow:visible;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;white-space:pre-wrap">#<span style="box-sizing:border-box">define</span> <span style="box-sizing:border-box">RTE_MAX_TAILQ</span> <span style="box-sizing:border-box">32</span></td></tr><tr style="box-sizing:border-box"><td id="m_3847235289724525773gmail-L39" style="box-sizing:border-box;padding:0px 10px;width:50px;min-width:50px;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;line-height:20px;text-align:right;white-space:nowrap;vertical-align:top"></td></tr></tbody></table><br></div><div><br></div><div>I could not find a good documentation on how to calculate the proper values for these parameters</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 28, 2022 at 4:10 PM Dmitry Kozlyuk <<a href="mailto:dmitry.kozliuk@gmail.com" target="_blank">dmitry.kozliuk@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2022-07-28 15:05 (UTC+0300), MOD:<br> > Hi, Thanks for the response!<br> > the DPDK version is 20.11.4<br> > <br> > the stack trace is:<br> > malloc_elem_can_hold() // librte_eal.so.21<br> > find_suitable_element() // librte_eal.so.21<br> > malloc_heap_alloc() // librte_eal.so.21<br> > rte_memzone_reserve_thread_safe() // librte_eal.so.21<br> > rte_mempool_populate_default() // librte_mempool.so.21<br> > rte_mempool_create() // librte_mempool.so.21<br> <br> Is this all the info---no arguments, no lines?<br> You're using a debug build of DPDK, right?<br> <br> > RTE_MALLOC_DEBUG doesn't seem to change anything,<br> > but I noticed that I have been wrong about the allocation succeeding<br> > (not because of RTE_MALLOC_DEBUG)<br> > <br> > the error happens right on the first attempt.<br> <br> Did you try running with ASAN (meson -Db_sanitize=address)?<br> <br> Can you provide a short code to reproduce<br> or does it happen only in a larger program?<br> <br> Please keep Cc: <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> so that more people can join if they want.<br> </blockquote></div> </blockquote></div>