[PATCH v2 1/6] eal: add static per-lcore memory allocation facility

[Mattias Rönnblom]

On 2024-09-18 12:11, Jerin Jacob wrote:
> On Thu, Sep 12, 2024 at 8:52 PM Jerin Jacob <jerinjacobk at gmail.com> wrote:
>>
>> On Thu, Sep 12, 2024 at 7:11 PM Morten Brørup <mb at smartsharesystems.com> wrote:
>>>
>>>> From: Jerin Jacob [mailto:jerinjacobk at gmail.com]
>>>> Sent: Thursday, 12 September 2024 15.17
>>>>
>>>> On Thu, Sep 12, 2024 at 2:40 PM Morten Brørup <mb at smartsharesystems.com>
>>>> wrote:
>>>>>
>>>>>> +#define LCORE_BUFFER_SIZE (RTE_MAX_LCORE_VAR * RTE_MAX_LCORE)
>>>>>
>>>>> Considering hugepages...
>>>>>
>>>>> Lcore variables may be allocated before DPDK's memory allocator
>>>> (rte_malloc()) is ready, so rte_malloc() cannot be used for lcore variables.
>>>>>
>>>>> And lcore variables are not usable (shared) for DPDK multi-process, so the
>>>> lcore_buffer could be allocated through the O/S APIs as anonymous hugepages,
>>>> instead of using rte_malloc().
>>>>>
>>>>> The alternative, using rte_malloc(), would disallow allocating lcore
>>>> variables before DPDK's memory allocator has been initialized, which I think
>>>> is too late.
>>>>
>>>> I thought it is not. A lot of the subsystems are initialized after the
>>>> memory subsystem is initialized.
>>>> [1] example given in documentation. I thought, RTE_INIT needs to
>>>> replaced if the subsystem called after memory initialized (which is
>>>> the case for most of the libraries)
>>>
>>> The list of RTE_INIT functions are called before main(). It is not very useful.
>>>
>>> Yes, it would be good to replace (or supplement) RTE_INIT_PRIO by something similar, which calls the list of "INIT" functions at the appropriate time during EAL initialization.
>>>
>>> DPDK should then use this "INIT" list for all its initialization, so the init function of new features (such as this, and trace) can be inserted at the correct location in the list.
>>>
>>>> Trace library had a similar situation. It is managed like [2]
>>>
>>> Yes, if we insist on using rte_malloc() for lcore variables, the alternative is to prohibit establishing lcore variables in functions called through RTE_INIT.
>>
>> I was not insisting on using ONLY rte_malloc(). Since rte_malloc() can
>> be called before rte_eal_init)(it will return NULL). Alloc routine can
>> check first rte_malloc() is available if not switch over glibc.
> 
> 
> @Mattias Rönnblom This comment is not addressed in v7. Could you check?

Calling rte_malloc() and depending on it returning NULL if it's too 
early in the initialization process sounds a little fragile, but maybe 
it's fine.

One issue with lcore-variables-in-huge-pages I've failed to mentioned 
this time around this is being discussed is that it would increase 
memory usage by something like RTE_MAX_LCORE * 0.5 MB (or more probably 
a little more).

In the huge pages case, you can't rely on demand paging to avoid 
bringing in unused pages.

That said, I suspect some very latency-sensitive apps lock all pages in 
memory, and thus lose out on this OS feature.

I suggest we just leave the first incarnation of lcore variables in 
normal pages.

Thanks for the reminder.