[dpdk-users] lcore stack location and NUMA nodes

[Stephen Hemminger]

On Tue, 14 May 2019 14:50:00 +0000
"N. Benes" <nbenes at eso.org> wrote:

> Hi all,
> 
> is it ensured that the stacks of lcore threads with set affinity are
> located on the same NUMA node as the assigned CPU core?

No stacks are not directly affinitied to NUMA node.
Stacks live in regular (virtual) memory and the operating system manages
VM for processes/threads. The Linux MM subsystem may try and choose a page
that is on the same node as where thread is scheduled.

You can't DMA off of the stack and stack is highly cached data so
so it shouldn't matter much for a normal application.

> 
> So far, I understand that e.g. `rte_eal_mp_remote_launch()` does not
> actually create threads but just communicates the function to execute to
> already existing threads (via pipes). These threads are created during
> initialisation of the EAL:
> 
> https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal.c#n1164
> 
> The threads execute a state machine `eal_thread_loop()` where the
> affinity is changed/set for the corresponding lcore/thread:
> 
> https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal_thread.c#n119
> 
> However, couldn't this lead to the stacks of some threads to be still
> located on the (different) NUMA node where `pthread_create()` was called
> from, i.e. possible decreased performance?
> I did not find an explicit relocation of thread stacks.
> (I also did not find anything related in the glibc/nptl code)
> 
> One possible fix may be to change the affinity of the main/master thread
> before calling `pthread_create()` for the slaves and to reset the
> affinity of the master thread after all slave threads are created.
> 
> Cheers!