DPDK 22.11.2 requires too much VIRT memory, how to reduce

[Lombardo, Ed]

Hi,
I am revisiting this issue I have been living with.  The workaround I am using is to not memlock memory in our application.  
Error is "VIRT memory is too high and mmap fails, Cannot allocate memory (12)."

I tried DPDK 23.11 and I see the same issue I see with DPDK 22.11.  
In meson I tried setting -Db_lto=true -Dbuildtype=minsize but did not help my issue.

We have only 16 GB of memory and I setup 2x1GB=2GB hugepage size, (legacy mode EAL setting).
Also running Oracle91 kernel 5.14.0-284.

I see same the issue on virtual VM on Intel host, and on hardware bare metal with Atom processor.  (Both running Oracle91)

Can I reduce VIRT memory if I switch from static libraries to shared libraries?

Any help will be greatly appreciated.

Thanks,
Ed

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk at gmail.com> 
Sent: Friday, November 10, 2023 4:38 AM
To: Lombardo, Ed <Ed.Lombardo at netscout.com>
Cc: users <users at dpdk.org>
Subject: Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

2023-11-10 12:31 (UTC+0300), Dmitry Kozlyuk:
> Hi Ed,
> 
> 2023-11-10 00:16 (UTC+0000), Lombardo, Ed:
> > I finally finished testing all the options and found the VIRT value 
> > can be reduced from 66 GB to 16 GB with --legacy-mem setting in EAL init argument.
> 
> Right.
> By default, DPDK can use up to 64 GB of hugepage memory, so it 
> reserves 64 GB of VIRT (but does not map most of it), RES should be 
> low until the app actually allocates something.
> In legacy mode, DPDK maps all available hugepage memory at startup, in 
> your case 16 GB, VIRT and RES should be close.
> 
> >  So I therefore had to increase the VM memory from 16 GB to 24 GB 
> > (instead of 80 GB without this setting).
> 
> I don't understand why you have to do that.
> Possible VIRT is not limited by available RAM.
> DPDK should be able to reserve 64 GB of VIRT on a machine with 16 GB 
> of RAM, it will just be unable to map more than 16 GB (obviously).

Sorry, I've sent the message early by mistake.

> > I wonder what do we give up with this setting?

Most importantly, in legacy mode DPDK will consume all available hugepages at startup and will not free them back to the system until the all is terminated.
The default dynamic mode allocates and frees physical RAM on demand.
Some advanced DPDK memory API don't work in legacy mode.

> > 
> > All the other settings I tried and combinations of these had no 
> > impact (socket-limit=2048, single-file-segments, no-shconf, and 
> > no-telemetry) on VIRT memory.

Right, they should not.
DPDK assumes that VIRT reservation is almost free and unlimited.
May it be that your system somehow limits it?