[dpdk-dev] Big spike in DPDK VSZ

[Burakov, Anatoly]

On 04-Feb-20 12:07 PM, siddarth rai wrote:
> Hi Anatoly,
> 
> You mentioned that the maximum size of mempool is limited.
> Can you tell what is the limit and where is it specified?
> 
> Regards,
> Siddarth
> 

The mempool size itself isn't limited.

However, due to the nature of how memory subsystem works, there is 
always an upper limit to how much memory you can reserve (because once 
you run out of pre-reserved address space, you will effectively run out 
of memory).

Given that the current hardcoded upper limit for reserved address space 
is 128GB, this is effectively the limit you are referring to - you won't 
be able to create a mempool larger than 128GB because there is not 
enough reserved address space in which to put a bigger mempool.

The same applies to any other memory allocation, with a caveat that 
while a mempool can be non-contiguous in terms of VA memory (i.e. 
consist of several discontiguous VA areas), a single unit of memory 
allocation (i.e. a call to rte_malloc or rte_memzone_create) can only be 
as big as the largest contiguous chunk of address space.

Given that the current hardcoded limit is 16GB for 2M pages, and 32GB 
for 1GB pages, each allocation can be at most 16GB or 32GB long, 
depending on the page size from which you are allocating.

So, on a system with 1G and 2M pages, a mempool can only be as big as 
128GB, and each individual chunk of memory in that mempool can only be 
as big as 16GB for 2M pages, or 32GB for 1G pages.

These are big numbers, so in practice no one hits these limitations. 
Again, this is just the price you have to pay for supporting dynamic 
memory allocation in secondary processes. There is simply no other way 
to guarantee that all shared memory will reside in the same address 
space in all processes.

Notably, device hotplug doesn't provide such guarantee, which is why 
device hotplug (or even initialization) can fail in a secondary process. 
However, in device hotplug i suppose this is acceptable (or the 
community thinks it is). In the memory subsystem, i chose to be 
conservative and to always guarantee correctness, at a cost of placing 
an upper limit on memory allocations.

-- 
Thanks,
Anatoly