<div dir="ltr">Hi <a class="gmail_plusreply" id="plusReplyChip-0" href="mailto:stephen@networkplumber.org" tabindex="-1">@Stephen Hemminger</a> ,<div><br></div><div>We debugged further on this issue ..</div><div><br></div><div>We see that position and entry_size(position * key_entry_size) crosses above int32_t it overflows and it will corrupt random index of hash table.</div><div><br></div><div>in Nut shell hash table is only supporting upto 2topower32 size entries on the table. </div><div><br></div><div>Below mentioned snapshot of the code change.</div><div><br></div><div>Thanks,</div><div>Venkatesh.</div><div><br></div><div><br></div><div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)">    struct rte_hash_key *k, *keys = h->key_store;</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-63" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">63</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus  </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span>-    k = (struct rte_hash_key *) ((char *) keys + (position + 1) *</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-64" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">64</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus  </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span>-                                    h->key_entry_size);</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-65" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">65</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus  </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span>+ k = (struct rte_hash_key *) ((char *) keys + (uint64_t) (position + 1) *</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-66" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">66</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus  </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span>+                                 (uint64_t) h->key_entry_size);</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-67" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">67</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Plus  </span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-line-icon gmail-added" style="box-sizing:inherit"></span></span>       *key = k->key;</span></div><div class="gmail-repos-diff-contents-row gmail-monospaced-text" role="row" style="box-sizing:inherit;font-family:Menlo,Consolas,"Courier New",monospace;display:table-row;color:rgba(0,0,0,0.9);font-size:12px;white-space:pre"><span role="cell" class="gmail-repos-add-comment-widget gmail-text-center" style="box-sizing:inherit;text-align:center;display:table-cell;min-width:16px;min-height:16px"><span id="gmail-m-diff-widget-dpdk-uint64-fix2.patch-68" tabindex="0" role="button" style="box-sizing:inherit"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">Add comment</span></span></span><span role="cell" class="gmail-padding-horizontal-8 gmail-text-right gmail-secondary-text" style="box-sizing:inherit;color:rgba(0,0,0,0.55);text-align:right;padding-left:8px;padding-right:8px;display:table-cell"><span class="gmail-screen-reader-only" style="box-sizing:inherit;height:1px;width:1px;overflow:hidden;padding:0px;margin:0px">68</span><span aria-hidden="true" style="box-sizing:inherit"><span class="gmail-repos-line-number" style="box-sizing:inherit"></span></span></span><span aria-hidden="true" class="gmail-repos-collapsed-comment" style="box-sizing:inherit;display:table-cell;min-width:20px"></span><span tabindex="-1" class="gmail-padding-right-8 gmail-repos-line-content gmail-added" role="cell" style="box-sizing:inherit;padding-right:8px;width:680.969px;display:table-cell;background-color:rgba(51,153,51,0.1)">

</span></div></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Tue, Jun 10, 2025 at 12:19 PM venkatesh bs <<a href="mailto:venki.bsv@gmail.com">venki.bsv@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi <a class="gmail_plusreply" id="m_2940332444406401942gmail-plusReplyChip-0" href="mailto:stephen@networkplumber.org" target="_blank">@Stephen Hemminger</a> ,<div><br></div><div>Thanks for the reply , please find the details below.</div><div><br></div><div>DPDK version : 20.11.6</div><div><br>#define LOADBAL_HASH_ENTRIES_MAX (1024*1024*36)<br>#define LOADBAL_HASH_TABLE_SIZE_MULTIPLIER 2<br><br>hashSizeMultiplier = LOADBAL_HASH_TABLE_SIZE_MULTIPLIER;<br><br>  struct rte_hash_parameters loadbal_hash_params = {<br>                  .name = NULL,<br>                  .entries = LOADBAL_HASH_ENTRIES_MAX * hashSizeMultiplier,<br>                  .key_len = sizeof(flow_key_t),<br>                  .hash_func = app_hash_crc(internally it calls rte_hash_crc_4byte for v4/v6)<br>                  .hash_func_init_val = 0,<br>              };<br><br>First, what is the return value, which error?<br>       we captured only the return value, and will check what  error it is returning.<br><br>IPv4 Load-Bal Flow hash table:<br>    numInsertions:          998214247<br>    numInsertionsFailures:  4252<br>    numRemovals:            997197485<br>    numRemovalFailures:     214902<br>    numObjects:             1016762<br>    NumBuckets:             75497472<br>    TableCapacity:          75497472<br>    LoadFactor:             1%<br>    NumLookupSuccess:       389165605<br>    NumLookupFails:         2704814006<br>    Failure Analysis:<br>      TableFull(>=95%):    0<br>      HighLoad(75-95%):    0<br>      MediumLoad(50-75%):  0<br>      LowLoad(<50%):       4252<br>   </div></div>We are try to analyze the code and find out the details,<div>Initially there will be no issues, insertion starts failing after 24 HRS or so.</div><div><br></div><div>Thanks,</div><div>Venkatesh.</div><div><br><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jun 8, 2025 at 9:17 PM Stephen Hemminger <<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sun, 8 Jun 2025 00:36:02 +0530<br>
venkatesh bs <<a href="mailto:venki.bsv@gmail.com" target="_blank">venki.bsv@gmail.com</a>> wrote:<br>
<br>
> Hi @dpdk community,<br>
> <br>
> In my application, I am using the dpdk hash table as below.<br>
> <br>
> 1. calculate the hash value.<br>
> 2. using this signature add the key and data into  a table.<br>
> 3. lock is used in the 2nd call (add).<br>
> <br>
> Below is the code sniffer for the same.<br>
> ==================================================================<br>
>      hash_sig_t sig = rte_hash_hash(hash_tablele, (void *) &new_key);<br>
> <br>
>      pthread_mutex_lock(&lock);<br>
>       int32_t ret = rte_hash_add_key_with_hash_data(hash_table,<br>
>                           (void *)&new_key,<br>
>                            sig,<br>
>                            info);<br>
> <br>
> if (ret < 0)<br>
> {<br>
>          pthread_mutex_unlock(lock);<br>
>          return TOS_E_FAIL;<br>
> }<br>
> pthread_mutex_unlock(lock);<br>
> <br>
> return OK..<br>
> ==================================================================<br>
> <br>
> My application is having a lot of threads and when run with heavy load , I<br>
> am getting a lot of insertion failure, i felt the reason could be<br>
> calculating and adding the has value in 2 api's as opposed to<br>
> rte_hash_add() that is safe under a lock..<br>
> <br>
> <br>
> Please let me know your thoughts.<br>
<br>
Which architecture and DPDK version?<br>
What flags did you use during hash creation?<br>
<br>
As always with open source, the first thing to do is look at the source<br>
and see what is really happening, rather than just relying on the documentation.<br>
<br>
I assume you are using the default hash function which is CRC.<br>
<br>
First what is the return value, which error?<br>
It might just be key collisions. How big is the table and how many inserts?<br>
<br>
Also, the current DPDK hash flags with better locking and RCU.<br>
This would be faster than simple pthread mutex.<br>
<br>
</blockquote></div></div></div></div>
</blockquote></div>