cache miss increases when change rx descriptor from 512 to 2048

[Stephen Hemminger]

On Thu, 9 Feb 2023 03:58:56 +0000
"Xiaoping Yan (NSB)" <xiaoping.yan at nokia-sbell.com> wrote:

> Hi experts,
> 
> I had a traffic throughput test for my dpdk application, with same software and test case, only difference is the number of rx/tx descriptor:
> Rx/tx descriptor 512, test result 3.2mpps
> Rx/tx descriptor 2048, test result 3mpp
> From perf data, rx descriptor 2048 case has more cache miss, and lower instruction per cycle
> Perf for 512 rx descriptor
>       114289237792      cpu-cycles
>       365408402395      instructions              #    3.20  insn per cycle
>        74186289932      branches
>           36020793      branch-misses             #    0.05% of all branches
>         1298741388      bus-cycles
>            3413460      cache-misses              #    0.723 % of all cache refs
>          472363654      cache-references
> Perf for 2048 rx descriptor:
>        57038451185      cpu-cycles
>       173805485573      instructions              #    3.05  insn per cycle
>        35289607389      branches
>           15418885      branch-misses             #    0.04% of all branches
>          648164239      bus-cycles
>           13170596      cache-misses              #    1.702 % of all cache refs
>          773765263      cache-references
> 
> I understand it means more rx descriptor somehow causes more cache miss and then less instruction per cycle, so lower performance.
> 
> Any one observe similar results?
> Any idea to mitigate (or investigate further) the impact? (we want to use 2048 to better tolerate some jitter/burst)
> Any comment?
> 
> Thank you.
> 
> Br, Xiaoping
> 

If number of RX descriptors is small, there is a higher chance that when the device driver
walks the descriptor table or is using the resulting mbuf that the data is still in cache.

With large number of descriptors, since descriptors are used LIFO the rx descriptor and mbuf
will not be in cache.