[dpdk-dev] very high VIRT memory usage

[Burakov, Anatoly]

On 08-Jun-20 12:03 PM, Francesco wrote:
> Hi all,
> I upgraded an old DPDK-based app which was using DPDK 17.11 to latest DPDK
> 20.05 and I noticed that if I look  at "top" I see that the VIRT memory
> taken by my application is now 256.1GB while before it was <1GB.
> 
> I've seen this same behavior with also "testpmd" example... is this a known
> issue with latest DPDK versions?
> Can I tweak some setting to have VIRT memory usage more or less similar to
> RSS ?
> 
> I forgot to add I'm working on Linux, Centos7
> 
> Thanks,
> Francesco Montorsi
> 

There was a discussion on this not too long ago, but i can't seem to 
find it for some reason. Anyway, long story short, that's not a bug, 
that's by design.

Since 18.11 (or 18.05 to be precise), there is a new memory subsystem in 
DPDK that allows growing and shrinking DPDK memory usage at runtime. 
That means, you can start with zero hugepages preallocated, and then 
allocate as you go, letting the memory subsystem decide how much memory 
you need.

The catch is that all of this hugepage memory is allocated into 
somewhere, some virtual address space. And *that* address space is 
preallocated at startup, to allow for secondary processes to duplicate 
primary process's address space exactly, and allow dynamic allocation of 
*shared* memory at runtime.

This memory will show up in top et al. but the truth is, it's zero cost, 
because it's anonymous memory. It isn't actually taking up any RAM. It 
will show up in dumps (20.05 has already fixed that issue, and the fixes 
will probably be backported to stable, including 18.11), so unless you 
have a very specific problem, i don't think that's anything you should 
be concerned about.

-- 
Thanks,
Anatoly