[PATCH] config: limit lcore variable maximum size to 4k

[Mattias Rönnblom]

On 2024-11-09 00:52, Morten Brørup wrote:
>> From: Mattias Rönnblom [mailto:hofors at lysator.liu.se]
>> Sent: Friday, 8 November 2024 23.23
>>
>> On 2024-11-08 20:53, Morten Brørup wrote:
>>>> From: Morten Brørup [mailto:mb at smartsharesystems.com]
>>>> Sent: Friday, 8 November 2024 19.35
>>>>
>>>>> From: David Marchand [mailto:david.marchand at redhat.com]
>>>>> Sent: Friday, 8 November 2024 19.18
>>>>>
>>>>> OVS locks all pages to avoid page faults while processing packets.
>>>
>>> It sounds smart, so I just took a look at how it does this. I'm not
>> sure, but it seems like it only locks pages that are actually mapped
>> (current and future).
>>>
>>
>> mlockall(MLOCK_CURRENT) will bring in the whole BSS, it seems. Plus all
>> the rest like unused parts of the execution stacks, the data section
>> and
>> unused code (text) in the binary and all libraries it has linked to.
>>
>> It makes a simple (e.g., a unit test) DPDK 24.07 program use ~33x more
>> residential memory. After lcore variables, the same MLOCK_CURRENT-ed
>> program is ~30% larger than before. So, a relatively modest increase.
> 
> Thank you for testing this, Mattias.
> What are the absolute numbers, i.e. in KB, to get an idea of the numbers I should be looking for?
> 

Hello world type program with static linking. Default DPDK config. x86_64.

DPDK version  MAX_LCORE_VAR   EAL params         mlock  RSS [MB]
22.11         -               --no-huge -m 1000  no     22
24.11         1048576         --no-huge -m 1000  no     22
24.11         1048576         --no-huge -m 1000  yes    1576
24.11         4096            --no-huge -m 1000  yes    1445
22.11         -               -                  yes    333*
24.11         1048576         -                  yes    542*
24.11         4096            -                  yes    411*

* Excluding huge pages

If you are more selective what libraries you bring in, the footprint 
will be lower. How large a fraction is effectively unavoidable, I don't 
know. The relative increase will depends on how much memory the 
application uses, obviously. The hello world app doesn't have any 
app-level state.

> I wonder why the footprint grows at all... Intuitively the same variables should consume approximately the same amount of RAM, regardless how they are allocated.
> Speculating...

lcore variables use malloc(), which in turn does not bring in memory 
pages unless they are needed. Much of the lcore buffer will be unused, 
and not resident. I covered this, including some example calculation of 
the space savings, in an earlier thread. It may be in the programmer's 
guide as well, I don't remember.

> The lcore_states were allocated through rte_calloc() and thus used some space in the already allocated hugepages, so they didn't add more pages to the footprint. But they do when allocated and initialized as lcore variables.
> The first lcore variable allocated/initialized uses RTE_MAX_LCORE (128) pages of 4 KB each = 512 KB total. It seems unlikely that adding 512 KB increases the footprint by 30 %.
> 

mlockall() brings in all currently-untouched malloc()ed pages, growing 
the set of residential pages.

>>
>> The numbers are less drastic, obviously, for many real-world programs,
>> which have large packet pools and other memory hogs.
> 
> Agree.
> However, it would be good to understand why switching to lcore variables has this effect on the footprint when using mlockall() like OVS.
>