[dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count

[Neil Horman]

On Sun, Jul 24, 2016 at 06:08:00PM +0000, Wiles, Keith wrote:
> 
> 
> Sent from my iPhone
> 
> > On Jul 21, 2016, at 1:34 PM, Neil Horman <nhorman at redhat.com> wrote:
> > 
> >> On Thu, Jul 21, 2016 at 03:22:45PM +0000, Wiles, Keith wrote:
> >> 
> >>> On Jul 21, 2016, at 10:06 AM, Neil Horman <nhorman at redhat.com> wrote:
> >>> 
> >>> On Thu, Jul 21, 2016 at 02:09:19PM +0000, Wiles, Keith wrote:
> >>>> 
> >>>>> On Jul 21, 2016, at 8:54 AM, Neil Horman <nhorman at tuxdriver.com> wrote:
> >>>>> 
> >>>>> On Wed, Jul 20, 2016 at 10:32:28PM +0000, Wiles, Keith wrote:
> >>>>>> 
> >>>>>>> On Jul 20, 2016, at 3:16 PM, Neil Horman <nhorman at tuxdriver.com> wrote:
> >>>>>>> 
> >>>>>>> On Wed, Jul 20, 2016 at 07:47:32PM +0000, Wiles, Keith wrote:
> >>>>>>>> 
> >>>>>>>>> On Jul 20, 2016, at 12:48 PM, Neil Horman <nhorman at redhat.com> wrote:
> >>>>>>>>> 
> >>>>>>>>> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
> >>>>>>>>>> 2016-07-20 13:09, Neil Horman:
> >>>>>>>>>>> From: Neil Horman <nhorman at redhat.com>
> >>>>>>>>>>> 
> >>>>>>>>>>> John Mcnamara and I were discussing enhacing the validate_abi script to build
> >>>>>>>>>>> the dpdk tree faster with multiple jobs.  Theres no reason not to do it, so this
> >>>>>>>>>>> implements that requirement.  It uses a MAKE_JOBS variable that can be set by
> >>>>>>>>>>> the user to limit the job count.  By default the job count is set to the number
> >>>>>>>>>>> of online cpus.
> >>>>>>>>>> 
> >>>>>>>>>> Please could you use the variable name DPDK_MAKE_JOBS?
> >>>>>>>>>> This name is already used in scripts/test-build.sh.
> >>>>>>>>> Sure
> >>>>>>>>> 
> >>>>>>>>>>> +if [ -z "$MAKE_JOBS" ]
> >>>>>>>>>>> +then
> >>>>>>>>>>> +    # This counts the number of cpus on the system
> >>>>>>>>>>> +    MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> >>>>>>>>>>> +fi
> >>>>>>>>>> 
> >>>>>>>>>> Is lscpu common enough?
> >>>>>>>>> I'm not sure how to answer that.  lscpu is part of the util-linux package, which
> >>>>>>>>> is part of any base install.  Theres a variant for BSD, but I'm not sure how
> >>>>>>>>> common it is there.
> >>>>>>>>> Neil
> >>>>>>>>> 
> >>>>>>>>>> Another acceptable default would be just "-j" without any number.
> >>>>>>>>>> It would make the number of jobs unlimited.
> >>>>>>>> 
> >>>>>>>> I think the best is just use -j as it tries to use the correct number of jobs based on the number of cores, right?
> >>>>>>> -j with no argument (or -j 0), is sort of, maybe what you want.  With either of
> >>>>>>> those options, make will just issue jobs as fast as it processes dependencies.
> >>>>>>> Dependent on how parallel the build is, that can lead to tons of waiting process
> >>>>>>> (i.e. more than your number of online cpus), which can actually hurt your build
> >>>>>>> time.
> >>>>>> 
> >>>>>> I read the manual and looked at the code, which supports your statement. (I think I had some statement on stack overflow and the last time I believe anything on the internet :-) I have not seen a lot of differences in compile times with -j on my system. Mostly I suspect it is the number of paths in the dependency, cores and memory on the system.
> >>>>>> 
> >>>>>> I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.
> >>>>>> 
> >>>>>> $ export RTE_TARGET=x86_64-native-linuxapp-gcc 
> >>>>>> 
> >>>>>> $ time make install T=${RTE_TARGET}
> >>>>>> real    0m59.445s user    0m27.344s sys    0m7.040s
> >>>>>> 
> >>>>>> $ time make install T=${RTE_TARGET} -j
> >>>>>> real    0m26.584s user    0m14.380s sys    0m5.120s
> >>>>>> 
> >>>>>> # Remove the x86_64-native-linuxapp-gcc
> >>>>>> 
> >>>>>> $ time make install T=${RTE_TARGET} -j 72
> >>>>>> real    0m23.454s user    0m10.832s sys    0m4.664s
> >>>>>> 
> >>>>>> $ time make install T=${RTE_TARGET} -j 8
> >>>>>> real    0m23.812s user    0m10.672s sys    0m4.276s
> >>>>>> 
> >>>>>> cd x86_64-native-linuxapp-gcc
> >>>>>> $ make clean
> >>>>>> $ time make
> >>>>>> real    0m28.539s user    0m9.820s sys    0m3.620s
> >>>>>> 
> >>>>>> # Do a make clean between each build.
> >>>>>> 
> >>>>>> $ time make -j
> >>>>>> real    0m7.217s user    0m6.532s sys    0m2.332s
> >>>>>> 
> >>>>>> $ time make -j 8
> >>>>>> real    0m8.256s user    0m6.472s sys    0m2.456s
> >>>>>> 
> >>>>>> $ time make -j 72
> >>>>>> real    0m6.866s user    0m6.184s sys    0m2.216s
> >>>>>> 
> >>>>>> Just the real time numbers in the following table.
> >>>>>> 
> >>>>>> processes     real Time   depdirs
> >>>>>>   no -j             59.4s        Yes
> >>>>>>     -j 8             23.8s        Yes
> >>>>>>    -j 72            23.5s        Yes
> >>>>>>      -j               26.5s        Yes
> >>>>>> 
> >>>>>>   no -j             28.5s         No
> >>>>>>     -j 8               8.2s         No
> >>>>>>    -j 72              6.8s         No
> >>>>>>      -j                 7.2s         No
> >>>>>> 
> >>>>>> Looks like the depdirs build time on my system:
> >>>>>> $ make clean -j
> >>>>>> $ rm .depdirs
> >>>>>> $ time make -j
> >>>>>> real    0m23.734s user    0m11.228s sys    0m4.844s
> >>>>>> 
> >>>>>> About 16 seconds, which is not a lot of savings. Now the difference from no -j to -j is a lot, but the difference between -j and -j <cpu_count> is not a huge saving. This leads me back to over engineering the problem when ‘-j’ would work just as well here.
> >>>>>> 
> >>>>>> Even on my MacBook Pro i7 system the difference is not that much 1m8s without depdirs build for -j in a VirtualBox with all 4 cores 8G RAM. Compared to 1m13s with -j 4 option.
> >>>>>> 
> >>>>>> I just wonder if it makes a lot of sense to use cpuinfo in this given case if it turns out to be -j works with the 80% rule?
> >>>>> It may, but that seems to be reason to me to just set DPDK_MAKE_JOBS=0, and
> >>>>> you'll get that behavior
> >>>> 
> >>>> Just to be sure, ‘make -j 0’ is not a valid argument to the -j option. It looks like you have to do ‘-j’ or ‘-j N’ or no option where N != 0
> >>>> 
> >>>> I think we just use -j which gets us the 80% rule and the best performance without counting cores.
> >>> Thats odd, specifying 0 works for me.  If it doesn't for you, specify $MAX_INT
> >>> or some other huge number would be comparable
> >> 
> >> rkwiles at supermicro (master):~/.../dpdk/x86_64-native-linuxapp-gcc$ make --version
> >> GNU Make 4.1
> >> Built for x86_64-pc-linux-gnu
> >> Copyright (C) 1988-2014 Free Software Foundation, Inc.
> >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law.
> >> 
> >> rkwiles at supermicro (master):~/.../dpdk/x86_64-native-linuxapp-gcc$ make -j 0
> >> make: the '-j' option requires a positive integer argument
> >> 
> >> rkwiles at supermicro (master):~/.../dpdk/x86_64-native-linuxapp-gcc$ lsb_release -a
> >> No LSB modules are available.
> >> Distributor ID:    Ubuntu
> >> Description:    Ubuntu 16.04.1 LTS
> >> Release:    16.04
> >> Codename:    xenial
> > I'm not saying your variant doesn't work, only that my copy of make does, but
> > its possible that I have some alternately patched version (I used to fix make
> > bugs way back when, so I may have an impure copy).  Regardless, my comment is
> > still valid, if you want to have unlimited jobs, you can just export
> > DPDK_MAKE_JOBS=<some very large number>
> 
> Neil
> 
Sorry for the delayed response, I've been on vacation.

> Your modified copy of make has no bearing on the topic we are taking about customers using dpdk in standard distros right?
> 
!?  I really don't know what you're saying here.  My only reason for commenting
on my copy of make was to consider an explination for why make -j 0 might be
working for me, and not for you.  I've since uninstalled and resinalled make on
my system, and my copy now behaves as yours does

But thats all really irrelevant.  I don't know what you mean by this not having
any bearing on the conversation since we're talking about customers using dpdk
in a distro.  We're not really talking about that at all (if we were using make
would be a moot point, since distro users tend to only use pre-compiled
binaries).  Were talking about accelerating the build process when comparing
ABIs on two different dpdk versions, thats all.

> Seems odd to me to send this out with 0 or lspci as it may fail because of no lspci and will fail on all Ubuntu systems. 
> 
Again, don't really understand what your saying.  If you look at the patch,
neither of your assertions are true.  With this patch and no other change, the
validate_abi script behaves exactly as it does now.  The only thing I've done is
add a check for the DPDK_MAKE_JOBS environment variable, and if its not set,
either:

a) Set DPDK_MAKE_JOBS to 1 if lscpu doesn't exist on the system
b) Set DPDK_MAKE_JOBS to the number of online cpus if lscpu does exist

All of that gets overridden if you manually set DPDK_MAKE_JOBS to something
else.

seems pretty straightforward to me.

> If we ship with 1 then why even bother the adding code and if I have to edit the file or some other method to get better compile performance then why bother as well.
> 
Please stop and think for a minute.  Why would you need to edit a file to change
anything?  If lscpu exists, then everything happens automatically.  If it
doesn't you can still just run:
export DPDK_MAKE_JOBS=$BIGNUMBER; validate_abi.sh <args>

and it works fine.  If ubuntu has some other utiilty besides lscpu to parse cpu
counts, fine, we can add that in as a feature, but I don't see why that should
stop other, non-ubuntu systems from taking advantage of this.

> Setting the value to some large number does not make any sense to me and if I have to edit file every time or maintain a patch just seems silly. 
> 
Goodness Keith, stop for just a minute with the editing the file train your on.
Its an environment variable, you don't have to edit a file to change it.

> It just seems easier to set it to -j and not use lspci at all. This way we all win as I am not following your logic at all.
> 
This doesn't even make any sense.  If you set it to -j then you get make issuing
jobs at the max rate make can issue them, which is potentially fine, but may not
be what developers want in the event they don't want this script to monopolize
their system.  The way its written now lets people get reasonable behavior in
most cases, and opt-in to the extreeme case should they desire.  That makes far
more sense to me, then just chewing up as much cpu as possible all the time.

Neil

> Keith
> > 
> > Neil
> > 
> >>> 
> >>> Neil
> >>> 
> >>>>> 
> >>>>> Neil
> >>>>> 
> >>>>>> On some other project with a lot more files like the FreeBSD or Linux distro, yes it would make a fair amount of real time difference.
> >>>>>> 
> >>>>>> Keith
> >>>>>> 
> >>>>>>> 
> >>>>>>> While its fine in los of cases, its not always fine, and with this
> >>>>>>> implementation you can still opt in to that behavior by setting DPDK_MAKE_JOBS=0
> >>>>>>> 
> >>>>>>> Neil
> >> 
>