<div dir="ltr">Hi <a class="gmail_plusreply" id="plusReplyChip-0" href="mailto:stephen@networkplumber.org" tabindex="-1">@Stephen Hemminger</a> ,<div><br></div><div>We debugged further on this issue ..</div><div><br></div><div>We see that position and entry_size(position * key_entry_size) crosses above int32_t it overflows and it will corrupt random index of hash table.</div><div><br></div><div>in Nut shell hash table is only supporting upto 2topower32 size entries on the table. </div><div><br></div><div>Below mentioned snapshot of the code change.</div><div><br></div><div>Thanks,</div><div>Venkatesh.</div><div><br></div><div><br></div><div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"> struct rte_hash_key *k, *keys = h->key_store;</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-63" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">63</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span>- k = (struct rte_hash_key *) ((char *) keys + (position + 1) *</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-64" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">64</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span>- h->key_entry_size);</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-65" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">65</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span>+ k = (struct rte_hash_key *) ((char *) keys + (uint64_t) (position + 1) *</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-66" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">66</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span>+ (uint64_t) h->key_entry_size);</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-67" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">67</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span> *key = k->key;</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-68" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">68</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)">
</span></div></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Tue, Jun 10, 2025 at 12:19 PM venkatesh bs <<a href="mailto:venki.bsv@gmail.com">venki.bsv@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi <a class="gmail_plusreply" id="m_2940332444406401942gmail-plusReplyChip-0" href="mailto:stephen@networkplumber.org" target="_blank">@Stephen Hemminger</a> ,<div><br></div><div>Thanks for the reply , please find the details below.</div><div><br></div><div>DPDK version : 20.11.6</div><div><br>#define LOADBAL_HASH_ENTRIES_MAX (1024*1024*36)<br>#define LOADBAL_HASH_TABLE_SIZE_MULTIPLIER 2<br><br>hashSizeMultiplier = LOADBAL_HASH_TABLE_SIZE_MULTIPLIER;<br><br> struct rte_hash_parameters loadbal_hash_params = {<br> .name = NULL,<br> .entries = LOADBAL_HASH_ENTRIES_MAX * hashSizeMultiplier,<br> .key_len = sizeof(flow_key_t),<br> .hash_func = app_hash_crc(internally it calls rte_hash_crc_4byte for v4/v6)<br> .hash_func_init_val = 0,<br> };<br><br>First, what is the return value, which error?<br> we captured only the return value, and will check what error it is returning.<br><br>IPv4 Load-Bal Flow hash table:<br> numInsertions: 998214247<br> numInsertionsFailures: 4252<br> numRemovals: 997197485<br> numRemovalFailures: 214902<br> numObjects: 1016762<br> NumBuckets: 75497472<br> TableCapacity: 75497472<br> LoadFactor: 1%<br> NumLookupSuccess: 389165605<br> NumLookupFails: 2704814006<br> Failure Analysis:<br> TableFull(>=95%): 0<br> HighLoad(75-95%): 0<br> MediumLoad(50-75%): 0<br> LowLoad(<50%): 4252<br> </div></div>We are try to analyze the code and find out the details,<div>Initially there will be no issues, insertion starts failing after 24 HRS or so.</div><div><br></div><div>Thanks,</div><div>Venkatesh.</div><div><br><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jun 8, 2025 at 9:17 PM Stephen Hemminger <<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sun, 8 Jun 2025 00:36:02 +0530<br>
venkatesh bs <<a href="mailto:venki.bsv@gmail.com" target="_blank">venki.bsv@gmail.com</a>> wrote:<br>
<br>
> Hi @dpdk community,<br>
> <br>
> In my application, I am using the dpdk hash table as below.<br>
> <br>
> 1. calculate the hash value.<br>
> 2. using this signature add the key and data into a table.<br>
> 3. lock is used in the 2nd call (add).<br>
> <br>
> Below is the code sniffer for the same.<br>
> ==================================================================<br>
> hash_sig_t sig = rte_hash_hash(hash_tablele, (void *) &new_key);<br>
> <br>
> pthread_mutex_lock(&lock);<br>
> int32_t ret = rte_hash_add_key_with_hash_data(hash_table,<br>
> (void *)&new_key,<br>
> sig,<br>
> info);<br>
> <br>
> if (ret < 0)<br>
> {<br>
> pthread_mutex_unlock(lock);<br>
> return TOS_E_FAIL;<br>
> }<br>
> pthread_mutex_unlock(lock);<br>
> <br>
> return OK..<br>
> ==================================================================<br>
> <br>
> My application is having a lot of threads and when run with heavy load , I<br>
> am getting a lot of insertion failure, i felt the reason could be<br>
> calculating and adding the has value in 2 api's as opposed to<br>
> rte_hash_add() that is safe under a lock..<br>
> <br>
> <br>
> Please let me know your thoughts.<br>
<br>
Which architecture and DPDK version?<br>
What flags did you use during hash creation?<br>
<br>
As always with open source, the first thing to do is look at the source<br>
and see what is really happening, rather than just relying on the documentation.<br>
<br>
I assume you are using the default hash function which is CRC.<br>
<br>
First what is the return value, which error?<br>
It might just be key collisions. How big is the table and how many inserts?<br>
<br>
Also, the current DPDK hash flags with better locking and RCU.<br>
This would be faster than simple pthread mutex.<br>
<br>
</blockquote></div></div></div></div>
</blockquote></div>