[PATCH] common/mlx5: fix high SMMU TLB miss with mempool alignment

[yangxingui]

Hi Slava,

Thanks for the review.

On 2026/8/6 15:56, Slava Ovsiienko wrote:
> Hi,
> 
> Why does cross-NUMA pinning cause the huge page size alignment?

In practice, scenarios involving intra-NUMA binding encounter issues 
with huge-page alignment.

Even with the loop bug (alignment block always entered), there is a
second issue: the if (reg_end < hugepage_end) gate uses strict <
instead of <=.
Runtime data with 1 GB hugepages on Kunpeng SoC + CX7 (NUMA1), MPRQ
enabled, 16 cores, 16 queues, queue-size 4096:
intra-NUMA (-l 40-55, all on NUMA1):
   reg_start       = 0x121b76f000
   reg_end         = 0x1240000000
   hugepage_start  = 0x1200000000
   hugepage_end    = 0x1240000000
Here reg_end == hugepage_end, so reg_end < hugepage_end is false.
Alignment is skipped, reg_start stays at 0x121b76f000 (not aligned to
the 1 GB boundary) → high TLB miss rate under SMMU.
cross-NUMA (-l 11-26, all on NUMA0, CX7 on NUMA1):
   reg_start       = 0x11d93b5000
   reg_end         = 0x11fdc46000
   hugepage_start  = 0x11c0000000
   hugepage_end    = 0x1200000000
Here reg_end (0x11fdc46000) < hugepage_end (0x1200000000) is true.
Alignment runs: reg_start → hugepage_start, reg_end → hugepage_end
→ MR is hugepage-aligned → no excessive TLB misses.
In both cases the mempool (~585 MB) fits within a single 1 GB hugepage
and reg_start is NOT aligned. The only difference is whether reg_end
happens to coincide with hugepage_end. The intra-NUMA case happens to
fill the hugepage to its exact end, which makes the < check fail.
The mempool is always allocated on NUMA1 via rte_mempool_create with
numa_node = dev->device->numa_node. We don't yet have a definitive
explanation for why the placement differs between the two pinning
configurations — it likely depends on the hugepage allocator's internal
state. But the key point is that alignment should not depend on this
coincidence; reg_start should be aligned regardless of where reg_end
falls.

> It seems we have a bug in the current code (and the patch fixes it somehow ) -
> the loop  "/* Check that all ranges are on pages of the same size. */"
> does not depend on iteration variable "I" and always runs till the end.
You are right about the loop bug — the original
loop always runs to completion because it uses msl->page_sz (fetched
once before the loop) instead of each range's own memseg list. Our
patch fixes this by calling rte_mem_virt2memseg_list() inside the loop.
> 
> The patch mostly looks good to me, there are just my 5 cents to improve:
> 
> 1.
>>    	msl = rte_mem_virt2memseg_list((void *)ranges[0].start);
> We could move this into the checking loop inside (prefertable). Or start the loop from i=1.
> Just to have rte_mem_virt2memseg_list() call in single point.
>>    	if (msl != NULL) {
>> -		uint64_t hugepage_sz = 0;
>> +		hugepage_sz = msl->page_sz;


> 2.
>> +	if (same_hugepage_sz && hugepage_sz > 0) {
>> +		unsigned int orig_ranges_n = ranges_n;
> No sense to run this code if hugepage_sz is equal to rte_mem_page_size (),
> all alignment and range merging is done by mlx5_mempool_get_chunks().
> 
> 3. Also, I would be careful about external memory. All ranges are already
> aligned to the correct page sizes, so let's add "if !is_extmem" checking.
> 

I will update the version based on your suggestions.

Thanks,
Xingui