[PATCH v2 0/3] allow easier use of high lcore-ids

[Bruce Richardson]

On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson at intel.com] Sent:
> > Monday, 7 April 2025 11.49
> > 
> > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > Hello Bruce,
> > >
> > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > <bruce.richardson at intel.com> wrote:
> > > >
> > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson wrote:
> > > > > Traditionally, DPDK has had a direct mapping of internal lcore-
> > ids, to
> > > > > the actual core numbers in use. With higher core count servers
> > becoming
> > > > > more prevalent the issue becomes one of increasing memory
> > footprint when
> > > > > using such a scheme, due to the need to have all arrays
> > dimensioned for
> > > > > all cores on the system, whether or not those cores are in use by
> > the
> > > > > app.
> > > > >
> > > > > Therefore, the decision was made in the past to not expand the
> > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was
> > recommended
> > > > > that users use the "--lcores" EAL parameter to take the high-
> > numbered
> > > > > cores they wish to use and map them to lcore-ids within the 0 -
> > 128
> > > > > range. While this works, this is a little clunky as it means that
> > > > > instead of just passing, for example, "-l 130-139", the user must
> > > > > instead pass "--lcores 0 at 130,1 at 131,2 at 132,3 at 133,...."
> > > > >
> > > > > This patchset attempts to simplify the situation by adding a new
> > flag to
> > > > > do this mapping automatically. To use cores 130-139 and map them
> > to ids
> > > > > 0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-
> > ids",
> > > > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > > > >
> > > > > Adding this new parameter required some rework of the existing
> > arg
> > > > > parsing code, because in current DPDK the args are parsed and
> > checked in
> > > > > the order they appear on the commandline. This means that using
> > the
> > > > > example above, the core parameter 130-139 will be rejected
> > immediately
> > > > > before the "map-lcore-ids" parameter is seen. To work around
> > this, the
> > > > > core (and service core) parameters are not parsed when seen,
> > instead
> > > > > they are only saved off and parsed after all arguments are
> > parsed. The
> > > > > "-l" and "-c" parameters are converted into "--lcores" arguments,
> > so all
> > > > > assigning of lcore ids is done there in all cases.
> > > > >
> > > > > RFC->v2: * converted printf to DEBUG log * added "-M" as shorter
> > > > > version of flag * added documentation * renamed internal API that
> > > > > was changed to avoid any potential
> > hidden
> > > > >   runtime issues.
> > > > >
> > > > > Bruce Richardson (3): eal: centralize core parameter parsing eal:
> > > > > convert core masks and lists to core sets eal: allow automatic
> > > > > mapping of high lcore ids
> > > > >
> > > > Ping for review.
> > > >
> > > > At a high level, does this feature seem useful to users?
> > >
> > > This seems useful, though I am not I would touch the existing
> > options.
> > > I would have gone with a simple -L option (taking the same kind of
> > > input than -l but with new behavior), and not combine a flag with
> > > existing options.
> > >
> > 
> > That would be an easier patchset to do up. However, it would then mean
> > that we have no less than 4 different ways to specify the cores to use:
> > "- c", "-l", "-L", "--lcores" - and therefore need 4 different sets of
> > parsing options for them, and more checks to ensure we have only one of
> > the 4 specified in any run. That's why I chose the modifier option, and
> > to try and consolidate the core setup a bit.
> > 
> > However, if having a completely new option is preferred, I am happy
> > enough to do up a different patchset for that.
> > 
> > > I scanned through the series, not much to say.  Maybe add a unit test
> > > for new cmdline option.
> > >
> > Sure. Once it's decided what approach (if any) to take, I'll do up a
> > new patchset, taking into account any relevant feedback on this set.
> > 
> > /Bruce
> 
> Changing the EAL parameter parser to a "two pass parser" makes sense.  I
> think checking for existence of more than one lcore specification options
> should suffice; we don't need to accept multiple lcore specification
> options and check for conflicts.
> 
> When remapping, do we need to support gaps in the "lcore" (logical cores)
> array, e.g. for secondary processes, or can it be continuous from 0 to
> the number of specified lcores?
> 
> And are new EAL parameters for this really beneficial?  Doesn't e.g. "-l
> 0-9 at 130-139,100 at 140" suffice?
> 
Actually, I believe "0-9 at 130-139"[1]  is not the same as
"0 at 130,1 at 131,2 at 132,...". The latter affinities one thread to one core ten
times over, while the former affinitizes 10 threads to 10 cores - leaving
those threads free to move about within the 10 cores specified.

Just to confirm, I tweaked our helloworld example to print the cpu affinity
of each core when printing.

./build/examples/dpdk-helloworld --no-pci --lcores '(0-3)@(30-33)'
EAL: Detected CPU lcores: 96
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /run/user/11304126/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
hello from core 1, with thread affinity for cores: 30 31 32 33 
hello from core 3, with thread affinity for cores: 30 31 32 33 
hello from core 2, with thread affinity for cores: 30 31 32 33 
hello from core 0, with thread affinity for cores: 30 31 32 33 

./build/examples/dpdk-helloworld --no-pci --lcores '0 at 30,1 at 31,2 at 32,3 at 33'
EAL: Detected CPU lcores: 96
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /run/user/11304126/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
hello from core 1, with thread affinity for cores: 31 hello from core 3, with thread affinity for cores: 
hello from core 2, with thread affinity for cores: 32 
hello from core 0, with thread affinity for cores: 30 
33 

Regards,
/Bruce

[1] This actually needs to be "(0-9)@(130-139)", and with "--lcores", not
just "-l", there are actually different flags with different behaviours