[dpdk-dev] [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts detection

[Juraj Linkeš]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson at intel.com>
> Sent: Tuesday, November 3, 2020 10:45 AM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli at arm.com>
> Cc: Juraj Linkeš <juraj.linkes at pantheon.tech>; Ruifeng Wang
> <Ruifeng.Wang at arm.com>; Phil Yang <Phil.Yang at arm.com>;
> vcchunga at amazon.com; Dharmik Thakkar <Dharmik.Thakkar at arm.com>;
> jerinjacobk at gmail.com; hemant.agrawal at nxp.com; dev at dpdk.org; nd
> <nd at arm.com>
> Subject: Re: [RFC PATCH v3 3/6] build: automatic NUMA and cpu counts
> detection
> 
> On Mon, Nov 02, 2020 at 07:01:44PM +0000, Honnappa Nagarahalli wrote:
> > <snip>
> >
> > > >
> > > Part of the confusion arises from the fact that originally that was
> > > the only parameter set by this - and on x86 it still is. Perhaps
> > > this parameter needs to
> > Just wondering, for x86, what does it mean if we set the max_num_cores and
> max_numa_nodes based on dynamic detection for 'native' build?
> > ISA still remains the same as before. But, the build might not work on
> machines with higher number of cores and numa nodes.
> > At the same time, the build also might not work on a machine with a different
> ISA. The users need to be aware that the target machine has the same ISA and
> same number of cores/numa nodes as the target machine.
> >
> Yes, that is a fair summary.
> 

There's also additional confusion in what the build should produce with no user input. You mentioned that builds from CI are used outside of CI and when a user compiles up a binary without settings these [cpu and numa counts] explicitly it should run on 95%+ of systems of that type (why not 100%?). These requirements/expectations suggest that generic settings should be the default (machine=generic so that CI builds are usable on machines with different ISA and then using generic defaults for other settings (we're talking mainly about cpu and numa counts, but it should apply to everything - there are more settings that need generic defaults for Arm)).
If we change the default to generic, then setting machine=native would only set the machine args and not change the behavior of other vars (i.e. use defaults for the build machine).

> > > be renamed to "isa-level" or "architecture-flag" or similar to
> > > reflect its meaning. This would then allow a new "machine" setting,
> > > which can be considered separately. The question then is how much
> > > that helps with the main issue under discussion, that of cores and numa node
> values.
> > If we rename it, we will have backward compatibility issue (i.e. 'native' build on
> x86 will have different meaning and whoever wants the original meaning, have
> to change to using this new name). Not sure about the complexity in meson
> scripts.
> >
> 
> Yep, it was just a thought to see if it could help in this situation.
> 

I also don't think renaming or repurposing the machine meson option is the way to go, not only because of backwards compatibility but also because the option sets RTE_MACHINE and machine args, which lines up with the name nicely. As a side note, I grepped RTE_MACHINE and I don't see it being used anywhere - do we need that config?
But this doesn't work for Arm, since setting just machine args doesn't make sense - setting that also necessitates setting of some other dpdk config as well.

What could help is new meson option. I'm working on adding an option where the user could specify the set of target configuration, or soc configuration. I could add the configuration for generic build as one of the available sets and then we'd be able to use that variable instead of machine for Arm builds.

> >
> > >
> > > > But, I think other DPDK specific parameters should also be considered.
> > > > For ex: RTE_MAX_LCORE should have a particular value for 'generic'
> > > > build in
> > > all the supported architectures. The value could be different for
> > > each architecture, but it is fixed for the 'generic' build for a given
> architecture.
> > > Otherwise, the 'generic' build might not run on all the machines of
> > > that architecture.
> > > >
> > > > Similarly, for 'native' build, is there any reason not to include
> > > > other DPDK
> > > parameters as part of the definition? IMO, 'native' should refer to
> > > the entire build machine, not just the ISA. i.e. build on the target machine.
> > > >
> > >
> > > While I understand the idea here, it is somewhat complicated by the
> > > fact that the meson options given in "meson_options.txt" cannot be
> > > set by meson code, which means that when we change the machine flag
> > > to "native" we can only use or ignore the user-provided lcores and
> > > numa nodes setting - we have no way to change them and reflect those
> > > changes back to the user. :-( This leads to the situation in the
> > > discussion in this thread, where we start needing all sorts of magic
> > > values to indicate use of machine-type defaults or detected defaults.
> > I am wondering why we need to take the max_num_cores and
> max_numa_nodes from the user? This option was not provided in the make build
> system. I ask this question because for 'generic' this has to be a static/known
> configuration. For cross builds, this info can come (or derived) from the cross
> build file.
> > Was it supposed to be used in conjunction with 'native' build?
> >
> 
> Well, it was configurable in the build config files same as all other DPDK build
> settings with make. When working first on meson, I felt it was a setting the user
> might be likely to want to tune, which is why I put it into the meson_options.txt
> and nobody suggested otherwise on review [which is the reason why many of
> the current options are the way they are :-)].
> 
> From my side, I have a couple of unknowns:
> 1. How big a difference in terms of memory use etc. of DPDK does it make by
>    having really big values for these core/numa counts? If there is not much
>    difference, then there is indeed little value in having them configurable
>    at all, and we should just use big defaults and be done with it.
> 2. If there is a noticable difference in these settings, how many users are
>    going to want to actually go to the trouble of tweaking these?
> 3. How big an effort is it to switch to having these settings made entirely
>    dynamic at runtime? Doing so would naturally make the need for these
>    settings completely go away.
> 
> With all that said, I'd be ok with a number of solutions. I'm ok to have these
> dropped as meson options and just have them specified in other ways, e.g.
> cross-file, or from meson.build files. [For x86, I'd tend towards having them
> defined in rte_config.h inside x86-specific ifdefs].
> Alternatively, I'm also happy enough with the proposed scheme here of allowing
> user override, with platform defaults using "0"-value and detection using "-1".
> 
> Regards,
> /Bruce