<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi, Shuaisong</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Thank youm for the v2, sorry for the review delay.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
The patch mostly looks good to me.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
My concern is this:</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
>>if (is_extmem)<br>
>> + goto out;</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
It changes the existing behavior for the external memory.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Before there was combining all the ranges within the single hugepage for the external memory also.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
With the patch applied it is not anymore.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
For the regular memory - we can accept your proposal.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
For the external memory - we should keep existing behavior, not skip combining within a single huge page.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
I would also elaborate this approach and combine the ranges within the next huge pages (if any), but</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
not combine the huge page itself (for the external memory).</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
With best regards,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Slava</div>
<div><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>From:</b> Shuaisong Yang <yangshuaisong@h-partners.com><br>
<b>Sent:</b> Tuesday, November 3, 2026 9:35 AM<br>
<b>To:</b> yangxingui <yangxingui@huawei.com>; Slava Ovsiienko <viacheslavo@nvidia.com>; david.marchand@redhat.com <david.marchand@redhat.com>; Dariusz Sosnowski <dsosnowski@nvidia.com>; Bing Zhao <bingz@nvidia.com>; Ori Kam <orika@nvidia.com>; Suanming Mou
<suanmingm@nvidia.com><br>
<b>Cc:</b> dev@dpdk.org <dev@dpdk.org>; stephen@networkplumber.org <stephen@networkplumber.org>; NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>; dmitry.kozliuk@gmail.com <dmitry.kozliuk@gmail.com>; david.marchand@redhat.com <david.marchand@redhat.com>;
fengchengwen@huawei.com <fengchengwen@huawei.com>; liuyonglong@huawei.com <liuyonglong@huawei.com>; kangfenglong@huawei.com <kangfenglong@huawei.com>; yangshuaisong <yangshuaisong@h-partners.com><br>
<b>Subject:</b> Re: [PATCH v2] common/mlx5: fix high SMMU TLB miss with mempool alignment
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-size: 11pt;">> Hi Slava,<br>
><br>
> Gentle ping on this patch, following up on the earlier pings from<br>
> Sep 11 and Oct 27.<br>
><br>
> In the v1 discussion you mentioned the patch mostly looked good to<br>
> you and gave three suggestions — all of them are incorporated in v2<br>
> (see the changelog):<br>
> <a href="https://patches.dpdk.org/project/dpdk/patch/20260811085231.2260761-1-yangxingui@huawei.com/" id="OWAe7ac1763-f872-6bc4-4c32-04f9f3cb249a" class="OWAAutoLink" data-auth="NotApplicable">
https://patches.dpdk.org/project/dpdk/patch/20260811085231.2260761-1-yangxingui@huawei.com/</a><br>
><br>
> Our replies to your v1 comments, including the root cause analysis<br>
> (reg_end == hugepage_end causing the alignment to be skipped), are<br>
> here:<br>
> <a href="https://patches.dpdk.org/project/dpdk/patch/20260612071434.2722918-1-yangxingui@huawei.com/" id="OWAca175ee0-03cc-ed7f-d78f-1b29e60eecee" class="OWAAutoLink" data-auth="NotApplicable">
https://patches.dpdk.org/project/dpdk/patch/20260612071434.2722918-1-yangxingui@huawei.com/</a><br>
><br>
> The patch fixes a ~30% performance regression with CX7 under SMMU<br>
> (intra-NUMA pinning on Kunpeng SoC). It has been pending for about<br>
> three months — we'd really appreciate your final review. Happy to<br>
> dig further into why the hugepage placement differs between pinning<br>
> configurations, or provide more test data, if that helps.<br>
><br>
> Thanks!<br>
> Shuaisong & Xingui<br>
<br>
On 2026/10/27 19:32, Shuaisong Yang wrote:<br>
> Hi mlx5 team, just a gentle ping on this patch when you have a moment. Thanks!<br>
><br>
> On 2026/9/11 14:33, yangxingui wrote:<br>
>> Kindly ping for review...<br>
>><br>
>> On 2026/8/11 16:52, Xingui Yang wrote:<br>
>>> From: Shuaisong Yang <yangshuaisong@h-partners.com><br>
>>><br>
>>> On Kunpeng SoC with mlx CX7, dpdk-l3fwd with intra-NUMA core pinning<br>
>>> under SMMU nonstrict/strict mode shows about 30% performance degradation<br>
>>> compared to cross-NUMA pinning. With SMMU disabled or passthrough mode,<br>
>>> intra-NUMA performs as expected (slightly better than cross-NUMA).<br>
>>><br>
>>> CX7 in NUMA1<br>
>>> NUMA node0 CPU(s): 0-39<br>
>>> NUMA node1 CPU(s): 40-79<br>
>>><br>
>>> intra-NUMA:<br>
>>> dpdk-l3fwd -l 40-55 -n 4 -a 0000:17:00.1,mprq_en=1 -- -p 0x1 -P \<br>
>>> --config='(0,0,40),(0,1,41),(0,2,42),(0,3,43),(0,4,44),\<br>
>>> (0,5,45),(0,6,46),(0,7,47),(0,8,48),(0,9,49),\<br>
>>> (0,10,50),(0,11,51),(0,12,52),(0,13,53),\<br>
>>> (0,14,54),(0,15,55)' \<br>
>>> --rx-queue-size=4096 --tx-queue-size=4096 --rx-burst=64<br>
>>><br>
>>> cross-NUMA:<br>
>>> dpdk-l3fwd -l 11-26 -n 4 -a 0000:17:00.1,mprq_en=1 -- -p 0x1 -P \<br>
>>> --config='(0,0,11),(0,1,12),(0,2,13),(0,3,14),(0,4,15),\<br>
>>> (0,5,16),(0,6,17),(0,7,18),(0,8,19),(0,9,20),\<br>
>>> (0,10,21),(0,11,22),(0,12,23),(0,13,24),\<br>
>>> (0,14,25),(0,15,26)' \<br>
>>> --rx-queue-size=4096 --tx-queue-size=4096 --rx-burst=64<br>
>>><br>
>>> The root cause is that under SMMU enabled mode, the mempool allocated<br>
>>> for intra-NUMA pinning is aligned to system page size instead of<br>
>>> hugepage size, while cross-NUMA pinning correctly uses hugepage size<br>
>>> alignment. This causes high TLB miss rates under SMMU.<br>
>>><br>
>>> Align all memory ranges to hugepage boundaries during mempool<br>
>>> registration to ensure hugepage_sz alignment, thereby reducing TLB<br>
>>> misses and fixing the intra-NUMA performance degradation.<br>
>>><br>
>>> Fixes: 690b2a88c2f7 ("common/mlx5: add mempool registration facilities")<br>
>>> Cc: stable@dpdk.org<br>
>>><br>
>>> Signed-off-by: Shuaisong Yang <yangshuaisong@h-partners.com><br>
>>> Signed-off-by: Xingui Yang <yangxingui@huawei.com><br>
>>> ---<br>
>>> v2:<br>
>>> - Address review comments from Slava:<br>
>>> - Call rte_mem_virt2memseg_list() in a single point inside the loop.<br>
>>> - Skip the alignment block when hugepage_sz equals rte_mem_page_size(),<br>
>>> since mlx5_mempool_get_chunks() has already done the alignment.<br>
>>> - Skip the alignment for external memory with early goto, since ranges<br>
>>> are already aligned to the correct page size.<br>
>>> ---<br>
>>> .mailmap | 1 +<br>
>>> drivers/common/mlx5/mlx5_common_mr.c | 72 +++++++++++++++++++---------<br>
>>> 2 files changed, 51 insertions(+), 22 deletions(-)<br>
>>><br>
>>> diff --git a/.mailmap b/.mailmap<br>
>>> index fcb3d1bb3f..c8ba648d3c 100644<br>
>>> --- a/.mailmap<br>
>>> +++ b/.mailmap<br>
>>> @@ -1999,3 +1999,4 @@ Zongyu Wu <wuzongyu1@huawei.com><br>
>>> Zorik Machulsky <zorik@amazon.com><br>
>>> Zyta Szpak <zyta@marvell.com> <zr@semihalf.com><br>
>>> Zyta Szpak <zyta@marvell.com> <zyta.szpak@semihalf.com><br>
>>> +Shuaisong Yang <yangshuaisong@h-partners.com><br>
>>> diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c<br>
>>> index aa2d5e88a4..f9e1ec7468 100644<br>
>>> --- a/drivers/common/mlx5/mlx5_common_mr.c<br>
>>> +++ b/drivers/common/mlx5/mlx5_common_mr.c<br>
>>> @@ -1524,7 +1524,11 @@ mlx5_get_mempool_ranges(struct rte_mempool *mp, bool is_extmem,<br>
>>> * @param[in] is_extmem<br>
>>> * Whether the pool is contains only external pinned buffers.<br>
>>> * @param[out] out<br>
>>> - * Receives memory ranges to register, aligned to the system page size.<br>
>>> + * Receives memory ranges to register. For non-external memory, ranges<br>
>>> + * are aligned to the hugepage size if all ranges are on hugepages<br>
>>> + * of the same size, otherwise aligned to the system page size.<br>
>>> + * For external memory, ranges are kept as returned by<br>
>>> + * mlx5_get_mempool_ranges() (already aligned to the page size).<br>
>>> * The caller must release them with free().<br>
>>> * @param[out] out_n<br>
>>> * Receives the number of @p out items.<br>
>>> @@ -1541,7 +1545,8 @@ mlx5_mempool_reg_analyze(struct rte_mempool *mp, bool is_extmem,<br>
>>> {<br>
>>> struct mlx5_range *ranges = NULL;<br>
>>> unsigned int i, ranges_n = 0;<br>
>>> - struct rte_memseg_list *msl;<br>
>>> + bool same_hugepage_sz = true;<br>
>>> + uint64_t hugepage_sz = 0;<br>
>>> if (mlx5_get_mempool_ranges(mp, is_extmem, &ranges, &ranges_n) < 0) {<br>
>>> DRV_LOG(ERR, "Cannot get address ranges for mempool %s",<br>
>>> @@ -1550,34 +1555,57 @@ mlx5_mempool_reg_analyze(struct rte_mempool *mp, bool is_extmem,<br>
>>> }<br>
>>> /* Check if the hugepage of the pool can be shared. */<br>
>>> *share_hugepage = false;<br>
>>> - msl = rte_mem_virt2memseg_list((void *)ranges[0].start);<br>
>>> - if (msl != NULL) {<br>
>>> - uint64_t hugepage_sz = 0;<br>
>>> + if (is_extmem)<br>
>>> + goto out;<br>
>>> + /* Check that all ranges are on pages of the same size. */<br>
>>> + for (i = 0; i < ranges_n; i++) {<br>
>>> + struct rte_memseg_list *range_msl =<br>
>>> + rte_mem_virt2memseg_list((void *)ranges[i].start);<br>
>>> +<br>
>>> + if (range_msl == NULL) {<br>
>>> + same_hugepage_sz = false;<br>
>>> + break;<br>
>>> + }<br>
>>> + if (hugepage_sz == 0) {<br>
>>> + hugepage_sz = range_msl->page_sz;<br>
>>> + } else if (range_msl->page_sz != hugepage_sz) {<br>
>>> + same_hugepage_sz = false;<br>
>>> + break;<br>
>>> + }<br>
>>> + }<br>
>>> + /*<br>
>>> + * Align ranges to the hugepage boundaries and merge adjacent ones.<br>
>>> + * If the entire pool fits in a single hugepage, the MR for this<br>
>>> + * hugepage can be shared across mempools.<br>
>>> + */<br>
>>> + if (same_hugepage_sz && hugepage_sz > rte_mem_page_size()) {<br>
>>> + unsigned int orig_ranges_n = ranges_n;<br>
>>> - /* Check that all ranges are on pages of the same size. */<br>
>>> for (i = 0; i < ranges_n; i++) {<br>
>>> - if (hugepage_sz != 0 && hugepage_sz != msl->page_sz)<br>
>>> - break;<br>
>>> - hugepage_sz = msl->page_sz;<br>
>>> + ranges[i].start = RTE_ALIGN_FLOOR(ranges[i].start,<br>
>>> + hugepage_sz);<br>
>>> + ranges[i].end = RTE_ALIGN_CEIL(ranges[i].end,<br>
>>> + hugepage_sz);<br>
>>> }<br>
>>> - if (i == ranges_n) {<br>
>>> - /*<br>
>>> - * If the entire pool is within one hugepage,<br>
>>> - * combine all ranges into one of the hugepage size.<br>
>>> - */<br>
>>> - uintptr_t reg_start = ranges[0].start;<br>
>>> - uintptr_t reg_end = ranges[ranges_n - 1].end;<br>
>>> - uintptr_t hugepage_start =<br>
>>> - RTE_ALIGN_FLOOR(reg_start, hugepage_sz);<br>
>>> - uintptr_t hugepage_end = hugepage_start + hugepage_sz;<br>
>>> - if (reg_end < hugepage_end) {<br>
>>> - ranges[0].start = hugepage_start;<br>
>>> + ranges_n = 1;<br>
>>> + for (i = 1; i < orig_ranges_n; i++) {<br>
>>> + if (ranges[ranges_n - 1].end >= ranges[i].start)<br>
>>> + ranges[ranges_n - 1].end =<br>
>>> + RTE_MAX(ranges[ranges_n - 1].end,<br>
>>> + ranges[i].end);<br>
>>> + else<br>
>>> + ranges[ranges_n++] = ranges[i];<br>
>>> + }<br>
>>> + if (ranges_n == 1) {<br>
>>> + uintptr_t hugepage_end = ranges[0].start + hugepage_sz;<br>
>>> +<br>
>>> + if (ranges[0].end <= hugepage_end) {<br>
>>> ranges[0].end = hugepage_end;<br>
>>> - ranges_n = 1;<br>
>>> *share_hugepage = true;<br>
>>> }<br>
>>> }<br>
>>> }<br>
>>> +out:<br>
>>> *out = ranges;<br>
>>> *out_n = ranges_n;<br>
>>> return 0;<br>
>>><br>
><br>
<br>
</div>
</body>
</html>