[dpdk-dev] [PATCH 2/2] devtools: add path to additional shared object files

Richardson, Bruce bruce.richardson at intel.com
Wed Dec 18 17:00:18 CET 2019



> -----Original Message-----
> From: David Marchand <david.marchand at redhat.com>
> Sent: Wednesday, December 18, 2019 3:32 PM
> To: Laatz, Kevin <kevin.laatz at intel.com>
> Cc: Ruifeng Wang <ruifeng.wang at arm.com>; Richardson, Bruce
> <bruce.richardson at intel.com>; Aaron Conole <aconole at redhat.com>; Michael
> Santana <maicolgabriel at hotmail.com>; Thomas Monjalon
> <thomas at monjalon.net>; Yigit, Ferruh <ferruh.yigit at intel.com>; Andrew
> Rybchenko <arybchenko at solarflare.com>; dev <dev at dpdk.org>; Gavin Hu
> <gavin.hu at arm.com>; Honnappa Nagarahalli <honnappa.nagarahalli at arm.com>;
> nd <nd at arm.com>
> Subject: Re: [dpdk-dev] [PATCH 2/2] devtools: add path to additional
> shared object files
> 
> On Wed, Dec 18, 2019 at 2:43 PM Laatz, Kevin <kevin.laatz at intel.com>
> wrote:
> >
> > On 18/12/2019 08:23, David Marchand wrote:
> > > On Wed, Dec 18, 2019 at 6:39 AM Ruifeng Wang <ruifeng.wang at arm.com>
> wrote:
> > >> librte_mempool_ring.so and librte_pmd_null.so are in 'drivers'
> folder.
> > >> Add 'drivers' into LD_LIBRARY_PATH so that testpmd can find and
> > >> make use of these shared libraries.
> > >>
> > >> Signed-off-by: Ruifeng Wang <ruifeng.wang at arm.com>
> > >> Reviewed-by: Gavin Hu <gavin.hu at arm.com>
> > >> ---
> > >>   devtools/test-null.sh | 2 +-
> > >>   1 file changed, 1 insertion(+), 1 deletion(-)
> > >>
> > >> diff --git a/devtools/test-null.sh b/devtools/test-null.sh index
> > >> f39af2c06..548de8113 100755
> > >> --- a/devtools/test-null.sh
> > >> +++ b/devtools/test-null.sh
> > >> @@ -20,7 +20,7 @@ if [ ! -f "$testpmd" ] ; then
> > >>   fi
> > >>
> > >>   if ldd $testpmd | grep -q librte_ ; then
> > >> -       export LD_LIBRARY_PATH=$build/lib:$LD_LIBRARY_PATH
> > >> +       export
> > >> + LD_LIBRARY_PATH=$build/drivers:$build/lib:$LD_LIBRARY_PATH
> > >>          libs='-d librte_mempool_ring.so -d librte_pmd_null.so'
> > >>   else
> > >>          libs=
> > >> --
> > >> 2.17.1
> > >>
> > > I'm surprised to see this.
> > > So far, the CI ran fine without it, so something is different in the
> > > environment.
> > >
> > > I can see that the RPATH entry disappeared from the testpmd binary.
> > > Xenial:
> > > # readelf -d build/app/dpdk-testpmd |grep RPATH
> > >   0x000000000000000f (RPATH)              Library rpath:
> > > [$ORIGIN/../lib:$ORIGIN/../drivers:XXXXXXXXXXXXX]
> > >
> > > (not sure what XXXX purpose is, but different topic)
> > >
> > > Bionic:
> > > # readelf -d build/app/dpdk-testpmd |grep RPATH
> >
> > It looks like RPATH just changed to be named RUNPATH in Bionic:
> >
> > # readelf -d build/app/dpdk-testpmd | grep R.*PATH
> >   0x000000000000001d (RUNPATH)            Library runpath:
> > [$ORIGIN/../lib:$ORIGIN/../drivers:XXXXXXXXXXXXX]
> 
> Did some experiment with some test program and .so of mine.
> TL;DR, if I understand correctly, RPATH on the binary applies to all
> lookups, even in a subsequent .so code.
> But RUNPATH only applies to the current ELF, meaning that the dlopen() in
> my intermediate .so does not get it.
> 
> dlopen() is called from librte_eal.so, and RUNPATH on testpmd is not
> enough.
> 
> 
> Details:
> [dmarchan at dmarchan plop]$ cat main.c
> extern void loader(void);
> 
> int main(int argc, char *argv[])
> {
>     loader();
>     return 0;
> }
> [dmarchan at dmarchan plop]$ cat loader/loader.c #include <dlfcn.h> #include
> <stdio.h>
> 
> void loader(void)
> {
>     if (dlopen("lib.so", RTLD_NOW) == NULL)
>         fprintf(stderr, "%s\n", dlerror()); }
> 
> [dmarchan at dmarchan plop]$ cat lib/lib.c
> #include <stdio.h>
> 
> __attribute__((constructor))
> static void plop(void)
> {
>     fprintf(stdout, "plop\n");
> }
> 
> 
> # no rpath/runpath
> [dmarchan at dmarchan plop]$ gcc -o lib/lib.so -Wall -Werror -shared -fPIC
> lib/lib.c [dmarchan at dmarchan plop]$ gcc -o loader/loader.so -Wall -Werror
> -shared -fPIC loader/loader.c -ldl [dmarchan at dmarchan plop]$ gcc -o main -
> Wall -Werror main.c loader/loader.so [dmarchan at dmarchan plop]$ readelf -d
> main |grep R.*PATH [dmarchan at dmarchan plop]$ ./main
> lib.so: cannot open shared object file: No such file or directory
> 
> # using rpath on final binary
> [dmarchan at dmarchan plop]$ gcc -o main -Wall -Werror main.c
> loader/loader.so -Wl,-rpath,loader:lib [dmarchan at dmarchan plop]$ readelf -
> d main |grep R.*PATH
>  0x000000000000000f (RPATH)              Library rpath: [loader:lib]
> [dmarchan at dmarchan plop]$ ./main
> plop
> 
> # using runpath on final binary
> [dmarchan at dmarchan plop]$ gcc -o main -Wall -Werror main.c
> loader/loader.so -Wl,-enable-new-dtag,-rpath,loader:lib
> [dmarchan at dmarchan plop]$ readelf -d main |grep R.*PATH
>  0x000000000000001d (RUNPATH)            Library runpath: [loader:lib]
> [dmarchan at dmarchan plop]$ ./main
> lib.so: cannot open shared object file: No such file or directory
> 
> # using runpath on loader
> [dmarchan at dmarchan plop]$ gcc -o loader/loader.so -Wall -Werror -shared -
> fPIC loader/loader.c -ldl -Wl,-enable-new-dtag,-rpath,lib
> [dmarchan at dmarchan plop]$ readelf -d loader/loader.so |grep R.*PATH
>  0x000000000000001d (RUNPATH)            Library runpath: [lib]
> [dmarchan at dmarchan plop]$ gcc -o main -Wall -Werror main.c
> loader/loader.so -Wl,-enable-new-dtag,-rpath,loader
> [dmarchan at dmarchan plop]$ readelf -d main |grep R.*PATH
>  0x000000000000001d (RUNPATH)            Library runpath: [loader]
> [dmarchan at dmarchan plop]$ ./main
> plop
> 
> 
> > > Adding Bruce and Kevin, as I think this is the same issue than:
> > > http://mails.dpdk.org/archives/dev/2019-December/153627.html
> > > Could it be a change in meson?
> >
> > Yes, looks like the same error to me.
> >
> > I'm not sure this is solely a meson issue, I often need to pass -d
> > librte_mempool_ring.so when using make builds too. Maybe I'm missing
> > something here?
> 
> In a make build directory, all libraries ends up in the same lib/
> directory and the test-null.sh script work with the current
> LD_LIBRARY_PATH.
> 
> 
> Ruifeng patch seems valid to me, but I would love some explanations in the
> commitlog.
> 
> --
> David Marchand

Really, if we are running a shared-library build, all sorts of problems are
to be expected unless we do an "install" to place the .so files in their correct
locations on the filesystem.

For example, on install, the drivers are copied to a drivers directory off the
lib folder, but then symlinked to lib too, so that e.g. the bus drivers can be
found as dependencies of the other drivers.

For running binaries within the build directory, I always recommend using a 
static build instead, to avoid all these issues - this is why static linking
is still the DPDK default.

In terms of this patch, I have no problem with it if it fixed the issue.

/Bruce


More information about the dev mailing list