[dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Bruce Richardson]

On Mon, Nov 02, 2020 at 07:01:44PM +0000, Honnappa Nagarahalli wrote:
> <snip>
> 
> > >
> > Part of the confusion arises from the fact that originally that was the only
> > parameter set by this - and on x86 it still is. Perhaps this parameter needs to
> Just wondering, for x86, what does it mean if we set the max_num_cores and max_numa_nodes based on dynamic detection for 'native' build?
> ISA still remains the same as before. But, the build might not work on machines with higher number of cores and numa nodes.
> At the same time, the build also might not work on a machine with a different ISA. The users need to be aware that the target machine has the same ISA and same number of cores/numa nodes as the target machine.
> 
Yes, that is a fair summary.

> > be renamed to "isa-level" or "architecture-flag" or similar to reflect its
> > meaning. This would then allow a new "machine" setting, which can be
> > considered separately. The question then is how much that helps with the
> > main issue under discussion, that of cores and numa node values.
> If we rename it, we will have backward compatibility issue (i.e. 'native' build on x86 will have different meaning and whoever wants the original meaning, have to change to using this new name). Not sure about the complexity in meson scripts.
> 

Yep, it was just a thought to see if it could help in this situation.

> 
> > 
> > > But, I think other DPDK specific parameters should also be considered.
> > > For ex: RTE_MAX_LCORE should have a particular value for 'generic' build in
> > all the supported architectures. The value could be different for each
> > architecture, but it is fixed for the 'generic' build for a given architecture.
> > Otherwise, the 'generic' build might not run on all the machines of that
> > architecture.
> > >
> > > Similarly, for 'native' build, is there any reason not to include other DPDK
> > parameters as part of the definition? IMO, 'native' should refer to the entire
> > build machine, not just the ISA. i.e. build on the target machine.
> > >
> > 
> > While I understand the idea here, it is somewhat complicated by the fact that
> > the meson options given in "meson_options.txt" cannot be set by meson
> > code, which means that when we change the machine flag to "native" we
> > can only use or ignore the user-provided lcores and numa nodes setting - we
> > have no way to change them and reflect those changes back to the user. :-(
> > This leads to the situation in the discussion in this thread, where we start
> > needing all sorts of magic values to indicate use of machine-type defaults or
> > detected defaults.
> I am wondering why we need to take the max_num_cores and max_numa_nodes from the user? This option was not provided in the make build system. I ask this question because for 'generic' this has to be a static/known configuration. For cross builds, this info can come (or derived) from the cross build file.
> Was it supposed to be used in conjunction with 'native' build?
> 

Well, it was configurable in the build config files same as all other DPDK
build settings with make. When working first on meson, I felt it was a
setting the user might be likely to want to tune, which is why I put it
into the meson_options.txt and nobody suggested otherwise on review [which
is the reason why many of the current options are the way they are :-)].

>From my side, I have a couple of unknowns:
1. How big a difference in terms of memory use etc. of DPDK does it make by
   having really big values for these core/numa counts? If there is not much
   difference, then there is indeed little value in having them configurable
   at all, and we should just use big defaults and be done with it.
2. If there is a noticable difference in these settings, how many users are
   going to want to actually go to the trouble of tweaking these?
3. How big an effort is it to switch to having these settings made entirely
   dynamic at runtime? Doing so would naturally make the need for these
   settings completely go away.

With all that said, I'd be ok with a number of solutions. I'm ok to have
these dropped as meson options and just have them specified in other ways,
e.g. cross-file, or from meson.build files. [For x86, I'd tend towards
having them defined in rte_config.h inside x86-specific ifdefs].
Alternatively, I'm also happy enough with the proposed scheme here of
allowing user override, with platform defaults using "0"-value and
detection using "-1".

Regards,
/Bruce