Dpdk allocates more memory, than available physically (hugepages)

[Dmitry Kozlyuk]

2023-02-08 03:43 (UTC+0100), Szymon Szozda:
> Hey,
> I'm running dpdk on a machine with 64GB of RAM. It is configured, so 16GiB
> (16 x 1GiB chunks) of hugepage memory is reserved on boot. I was expecting
> dpdk to consume only those 16GiB, but it seems it gets more than 30GiB of
> virtual memory ( I base it on memory VSZ output of top command ). The
> machine is 1 NUMA, 1 NIC. I did some debugging and I do not see any logic
> which limits the memory consumption, basically it seems that
> eal_dynmem_memseg_lists_init() will allocate the same amount, no matter how
> much RAM is physically available.
> 
> Is it expected? How to know that setup will not crash due to
> insufficient memory available? How to limit those memory consumption.by
> dpdk?

Hi,

DPDK always reserves a large chunk of virtual address space,
but this costs almost nothing and does not need to be limited.
Then DPDK maps and unmaps actual pages to those addresses as needed.
DPDK does not crash if it runs out of hugepages reserved in the system
but merely returns NULL from its allocation API (rte_malloc).
Real memory consumption can be limited with --socket-limit EAL option.
See also --socket-mem and -m to reserve hugepages at DPDK startup.