[dpdk-dev] performance degradation with fpic

[Bruce Richardson]

On Thu, Oct 22, 2020 at 01:17:16PM +0000, Ali Alnubani wrote:
> Hi Bruce,
> Sorry for the delayed response.
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson at intel.com>
> > Sent: Monday, October 19, 2020 4:02 PM
> > To: Ali Alnubani <alialnu at nvidia.com>
> > Cc: dev at dpdk.org; NBU-Contact-Thomas Monjalon
> > <thomas at monjalon.net>; Asaf Penso <asafp at nvidia.com>
> > Subject: Re: [dpdk-dev] performance degradation with fpic
> > 
> > On Mon, Oct 19, 2020 at 11:47:48AM +0000, Ali Alnubani wrote:
> > > Hi Bruce,
> > >
> > > > -----Original Message-----
> > > > From: Bruce Richardson <bruce.richardson at intel.com>
> > > > Sent: Friday, October 16, 2020 12:59 PM
> > > > To: Ali Alnubani <alialnu at nvidia.com>
> > > > Cc: dev at dpdk.org; NBU-Contact-Thomas Monjalon
> > <thomas at monjalon.net>;
> > > > Asaf Penso <asafp at nvidia.com>
> > > > Subject: Re: [dpdk-dev] performance degradation with fpic
> > > >
> > > > On Thu, Oct 15, 2020 at 06:08:04PM +0100, Bruce Richardson wrote:
> > > > > On Thu, Oct 15, 2020 at 04:00:44PM +0000, Ali Alnubani wrote:
> > > > > >    Hi Bruce,
> > > > > >
> > > > > >
> > > > > >    We have been seeing in some cases that the DPDK forwarding
> > > > performance
> > > > > >    is up to 9% lower when DPDK is built as static with meson compared
> > to a
> > > > > >    build with makefiles.
> > > > > >
> > > > > >
> > > > > >    The same degradation can be reproduced with makefiles on older
> > DPDK
> > > > > >    releases when building with EXTAR_CFLAGS set to “-fPIC”, it can also
> > be
> > > > > >    resolved in meson when passing “pic: false” to meson’s
> > static_library
> > > > > >    call (more tweaking needs to be done to prevent building shared
> > > > > >    libraries because this change breaks them).
> > > > > >
> > > > > >
> > > > > >    I can reproduce this drop with the following cases:
> > > > > >      * Baremetal / NIC: ConnectX-4 Lx / OS: RHEL7.4 / CPU: Intel(R)
> > > > > >        Xeon(R) Gold 6154. Testpmd command:
> > > > > >
> > > > > >    testpmd -c 0x7ffc0000 -n 4 -w d8:00.1 -w d8:00.0 --socket-
> > > > mem=2048,2048
> > > > > >    -- --port-numa-config=0,1,1,1 --socket-num=1 --burst=64 --txd=512
> > > > > >    --rxd=512 --mbcache=512 --rxq=2 --txq=2 --nb-cores=1
> > > > > > --no-lsc-
> > > > interrupt
> > > > > >    -i -a --rss-udp
> > > > > >      * KVM guest with SR-IOV passthrough / OS: RHEL7.4 / NIC:
> > > > > > ConnectX-5
> > > > /
> > > > > >        Host’s CPU: Intel(R) Xeon(R) Gold 6154. Testpmd command:
> > > > > >        testpmd --master-lcore=0 -c 0x1ffff -n 4 -w
> > > > > >        00:05.0,mprq_en=1,mprq_log_stride_num=6 --socket-
> > mem=2048,0 --
> > > > > >        --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=1024
> > > > > >        --rxd=1024 --mbcache=512 --rxq=16 --txq=16 --nb-cores=8
> > > > > >        --port-topology=chained --forward-mode=macswap --no-lsc-
> > > > interrupt
> > > > > >        -i -a --rss-udp
> > > > > >      * Baremetal / OS: Ubuntu 18.04 / NIC: ConnectX-5 / CPU: Intel(R)
> > > > > >        Xeon(R) CPU E5-2697A v4. Testpmd command:
> > > > > >        testpmd -n 4  -w 0000:82:00.0,rxqs_min_mprq=8,mprq_en=1  -w
> > > > > >        0000:82:00.1,rxqs_min_mprq=8,mprq_en=1 -c 0xff80  -- --
> > burst=64
> > > > > >        --mbcache=512 -i  --nb-cores=8  --rxq=8 --txq=8 --txd=1024
> > > > > >        --rxd=1024 --rss-udp --auto-start
> > > > > >
> > > > > >    The packets being received and forwarded by testpmd are of
> > IPv4/UDP
> > > > > >    type and 64B size.
> > > > > >
> > > > > >    Should we disable PIC in static builds?
> > > > > >
> > > > > >
> > > > >
> > > > > Hi Ali,
> > > > >
> > > > > thanks for reporting, though it's strange that you see such a big impact.
> > > > > In my previous tests with i40e driver I never noticed a difference
> > > > > between make and meson builds, and I and some others here have
> > > > > been using meson builds for any performance work for over a year
> > > > > now. That being said let me reverify what I see on my end.
> > > > >
> > > > > In terms of solutions, disabling the -fPIC flag globally implies
> > > > > that we can no longer build static and shared libs from the same
> > > > > sources, so we would need to revert to doing either a static or a
> > > > > shared library build but not both. If the issue is limited to only
> > > > > some drivers or some cases, we can perhaps add in a build option
> > > > > to have no-fpic-static builds, to be used in a cases where it is
> > problematic.
> > > > >
> > > > > However, at this point, I think we need a little more investigation.
> > > > > Is there any testing you can do to see if it's just in your
> > > > > driver, or in perhaps a mempool driver/lib that the issue appears,
> > > > > or if it's just a global slowdown? Do you see the impact with both clang
> > and gcc?
> > > > > I'll retest things a bit tomorrow on my end to see what I see.
> > > > >
> > > > Hi again,
> > > >
> > > > I've done a quick retest with the i40e driver on my system, using
> > > > the 20.08 version so as to have make vs meson direct comparison.
> > > > [For reference command used was: "sudo </path/to/testpmd>  -c F00000
> > > > -w af:00.0 -w
> > > > b1:00.0 -w da:00.0 -- --rxq=2 --txq=2 --rxd=2048 --txd=512" using
> > > > 3x40G ports to a single core running @3GHz.] No major performance
> > > > differences were seen, but if anything the meson build was very
> > > > slightly faster, as reported to Jerin, maybe 2%, though it's within the
> > margin of error.
> > > >
> > >
> > > Thanks for taking the time to investigate this.
> > >
> > > Disabling PIC for net/mlx5 driver alone in drivers/meson.build resolves the
> > issue for me.
> > > I saw this issue with gcc (tested with 4.8.5, 9.3.0, and 7.5.0). But I see now
> > that disabling PIC with an old clang version (clang 3.4.2, RHEL7.4) causes a
> > drop in performance, not an improvement like with gcc.
> > >
> > That's interesting.
> > 
> > When you just build with and without -fpic with newer clang, do you see the
> > same perf drop as with gcc? With the older clang, is the shared lib build faster
> > than the static one?
> 
> With the older clang on RHEL7.4, the shared lib is about ~2% slower compared to the static build.
> With clang 11 compiled from source on ubuntu 18.04, I'm getting good performance with static meson build, same performance as with makefiles with gcc, and ~6% better than the static meson gcc build. Disabling PIC on clang 11 degrades performance by ~4%.
> With clang 6.0.0 however, disabling PIC causes a very small drop (~0.1%).
> 
> This is on v20.08 with KVM ConnectX-5 SR-IOV passthrough. Command: "dpdk-testpmd --master-lcore=0 -c 0x1ffff -n 4 -w 00:05.0 --socket-mem=2048,0 -- --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=1024 --rxd=1024 --mbcache=512 --rxq=8 --txq=8 --nb-cores=4 --port-topology=chained --forward-mode=macswap --no-lsc-interrupt -i -a --rss-udp".
> 

So, am I right in saying that it appears the clang builds are all fine
here, that performance is pretty much as expected in all cases with the
default setting of PIC enabled? Therefore it appears that the issue is
limited to gcc builds at this point?

/Bruce