[PATCH 0/3] Defer lcore variables allocation

[Mattias Rönnblom]

On 2024-12-09 12:03, David Marchand wrote:
> Hello,
> 
> On Fri, Dec 6, 2024 at 12:02 PM Mattias Rönnblom <hofors at lysator.liu.se> wrote:
>>
>> On 2024-12-05 18:57, David Marchand wrote:
>>> As I had reported in rc2, the lcore variables allocation have a
>>> noticeable impact on applications consuming DPDK, even when such
>>> applications does not use DPDK, or use features associated to
>>> some lcore variables.
>>>
>>> While the amount has been reduced in a rush before rc2,
>>> there are still cases when the increased memory footprint is noticed
>>> like in scaling tests.
>>> See https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/2090931
>>>
>>
>> What this bug report fails to mention is that it only affects
>> applications using locked memory.
> 
> - By locked memory, are you referring to mlock() and friends?
> No ovsdb binary calls them, only the datapath cares about mlocking.
> 
> 
> - At a minimum, I understand the lcore var change introduced an
> increase in memory of 4kB * 128 (getpagesize() * RTE_MAX_LCORES),
> since lcore_var_alloc() calls memset() of the lcore var size, for
> every lcore.
> 

Yes, that is my understanding. It's also consistent with the 
measurements I've posted on this list.

> In this unit test where 1000 processes are kept alive in parallel,
> this means memory consumption increased by 512k * 1000, so ~500M at
> least.
> This amount of memory is probably significant in a resource-restrained
> env like a (Ubuntu) CI.
> 
> 

I wouldn't expect thousands of concurrent processes in a 
resource-constrained system. Sounds wasteful indeed. But sure, there may 
well be scenarios where this make sense.

> - I went and traced this unit tests on my laptop by monitoring
> kmem:mm_page_alloc, though there may be a better metrics when it comes
> to memory consumption.
> 
> # dir=build; perf stat -e kmem:mm_page_alloc -- tests/testsuite -C
> $dir/tests AUTOTEST_PATH=$dir/utilities:$dir/vswitchd:$dir/ovsdb:$dir/vtep:$dir/tests:$dir/ipsec::
> 2154
> 
> Which gives:
> - 1 635 489      kmem:mm_page_alloc for v23.11
> - 5 777 043      kmem:mm_page_alloc for v24.11
> 

Interesting. What is vm.overcommit_memory set to?

How much does process RSS differ between the two version?

This is what I get with a hello world type test program, which links to 
DPDK, but have not yet called rte_eal_init():

DPDK Version  Linking  RSS [kB]
22.11         Static   8448
23.11         Static   8192
9bbd44d638*   Static   8192
18b5049ab4**  Static   8704
24.11         Static   9472

22.11         Dynamic  2304
9bbd44d638*   Dynamic  2304
18b5049ab4**  Dynamic  2816
24.11         Dynamic  2816

No DPDK       -        1024

* 24.11rc commit before lcore variables patch set.
* Last commit in the initial lcore variables patch set.

Ubuntu 24.04 w/ all defaults. --no-huge was used.

In the dynamic case, you see the expected ~0.5 MB increase in foot print 
for yet-to-be-used lcore variables.

In the static case, you also see a ~0.5 MB bump with the lcore 
variables, and then another ~0.5 MB bump (for some other reason) until 
24.11.

(The dynamic hello world program only ends up being dynamically linked 
to very few of the DPDK shared objects, so ymmv).

> There is a 4M difference, where I would expect 128k.
> So something more happens, than a simple page allocation per lcore,
> though I fail to understand what.
> 
> 
> Btw, just focusing on lcore var, I did two more tests:
> - 1 606 998      kmem:mm_page_alloc for v24.11 + revert all lcore var changes.
> - 1 634 606      kmem:mm_page_alloc for v24.11 + current series with
> postponed allocations.
> 
> 

If one move initialization to shared object constructors (from having 
been at some later time), and then end up not running that 
initialization code at all (e.g., DPDK is not used), those code pages 
will increase RSS. That might well hurt more than the lcore variable 
memory itself, depending on how much code is run.

However, such read-only pages can be replaced with something more useful 
if the system is under memory pressure, so they aren't really a big 
issue as far as (real) memory footprint is concerned.

Just linking to DPDK (and its dependencies) already came with a 1-7 MB 
RSS penalty, prior to lcore variables. I wonder how much of that goes 
away if all RTE_INIT() type constructors are removed.