[dpdk-users] PANIC in rte_eal_config_reattach() with C++ app

Matczak, KrzysztofX krzysztofx.matczak at intel.com
Wed Nov 16 13:08:41 CET 2016
Previous message: [dpdk-users] vm2vm and host2vm performance using dpdk 16.11
Next message: [dpdk-users] PANIC in rte_eal_config_reattach() with C++ app
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi 

I'm developing C++ applicationthat links with DPDK 16.07 shared libraries.  The application itself attaches as secondary DPDK process to primary (eg testpmd) in order to get some statistics. 
There were almost no DPDK related problems on Fedora 23, but when I switched to Ubuntu 16.04 I  faced problems with mmap :

root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ./src/snap-collector-dpdkstats
EAL: Detected 28 lcore(s)
PANIC in rte_eal_config_reattach():
Cannot mmap memory for rte_config
8: [./src/snap-collector-dpdkstats() [0x408389]]
7: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ffff6464830]]
6: [./src/snap-collector-dpdkstats() [0x407f5c]]
5: [./src/snap-collector-dpdkstats() [0x4088db]]
4: [./src/snap-collector-dpdkstats() [0x418e49]]
3: [/usr/lib/librte_eal.so.2.1(rte_eal_init+0xe83) [0x7ffff70b4e83]]
2: [/usr/lib/librte_eal.so.2.1(__rte_panic+0xcb) [0x7ffff70b3a25]]
1: [/usr/lib/librte_eal.so.2.1(rte_dump_stack+0x2b) [0x7ffff70bc60b]]
Aborted (core dumped)


After many trial and error attempts, changing linking order, excluding or including dpdk libraries, rebuilding DPDK with  CONFIG_RTE_EAL_PMD_PATH variable set to pmd libraries location, I have come to conclusion: the only working workaround in my case is to run testpmd with explicitly set --base-virtaddr parameter. 
Please see detailed log of success and failure scenarios below. 
My questions are :
- anyone experienced similar problems with c++ app on Ubuntu 16.04 or any other platform ? Must note that problem is reproducible only with c++ app, other pure C  apps compiled and linked with gcc are working properly. Are there any other known solution for that problem ? 
- providing that -base-virtaddr is necessary for primary process (hovewer I doubt it) , how to determine it's value ?  Currently I'm setting it to --base-virtaddr=0x2aaa4a90000 which is value taken from some other mail thread, but when I run testpmd without it, I don't see any virtual address reported by EAL output.   


Here's my reproduction log: 

##================ setup  details ==================================================================
## kernel, OS, glibc versions
root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# uname -a
Linux silpixa00393941 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# lsb_release -r
Release:        16.04
root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ldd --version
ldd (Ubuntu GLIBC 2.23-0ubuntu4) 2.23
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

## DPDK  16.07 installation steps:
 * Modifications in `config/common_base`:
   #
   # Compile to share library
   #
   -CONFIG_RTE_BUILD_SHARED_LIB=n
   +CONFIG_RTE_BUILD_SHARED_LIB=y

   # Default driver path (or "" to disable)
   -CONFIG_RTE_EAL_PMD_PATH=""
   +CONFIG_RTE_EAL_PMD_PATH="/usr/lib/dpdk-pmd/"

 * Building target corresponding to underlying architecture, eg:
   make config T=x86_64-native-linuxapp-gcc
   make
   sudo make install prefix=/usr
 * Creation of  pmd folder and symlinks for pmd drivers
   mkdir -p /usr/lib/dpdk-pmd
   find /usr/lib -type f -name 'librte_pmd*' | while read path ; do ln -s $path /usr/lib/dpdk-pmd/`echo $path | grep -o 'librte_.*so'`; done

## make log and linked libraries of the secondary process (src/snap-collector-dpdkstats)  
root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# make
make  all-recursive
make[1]: Entering directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro'
Making all in src
make[2]: Entering directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro/src'
g++ -DHAVE_CONFIG_H -I. -I..    -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT snap_collector_dpdkstats-dpdkstats.o -MD -MP -MF .deps/snap_collector_dpdkstats-dpdkstats.Tpo -c -o snap_collector_dpdkstats-dpdkstats.o `test -f 'dpdkstats.cc' || echo './'`dpdkstats.cc
mv -f .deps/snap_collector_dpdkstats-dpdkstats.Tpo .deps/snap_collector_dpdkstats-dpdkstats.Po
g++ -DHAVE_CONFIG_H -I. -I..    -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT snap_collector_dpdkstats-dpdkstats_collector.o -MD -MP -MF .deps/snap_collector_dpdkstats-dpdkstats_collector.Tpo -c -o snap_collector_dpdkstats-dpdkstats_collector.o `test -f 'dpdkstats_collector.cc' || echo './'`dpdkstats_collector.cc
mv -f .deps/snap_collector_dpdkstats-dpdkstats_collector.Tpo .deps/snap_collector_dpdkstats-dpdkstats_collector.Po
g++ -DHAVE_CONFIG_H -I. -I..    -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT utils/snap_collector_dpdkstats-dpdk.o -MD -MP -MF utils/.deps/snap_collector_dpdkstats-dpdk.Tpo -c -o utils/snap_collector_dpdkstats-dpdk.o `test -f 'utils/dpdk.cc' || echo './'`utils/dpdk.cc
mv -f utils/.deps/snap_collector_dpdkstats-dpdk.Tpo utils/.deps/snap_collector_dpdkstats-dpdk.Po
g++ -DHAVE_CONFIG_H -I. -I..    -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT utils/snap_collector_dpdkstats-log.o -MD -MP -MF utils/.deps/snap_collector_dpdkstats-log.Tpo -c -o utils/snap_collector_dpdkstats-log.o `test -f 'utils/log.cc' || echo './'`utils/log.cc
mv -f utils/.deps/snap_collector_dpdkstats-log.Tpo utils/.deps/snap_collector_dpdkstats-log.Po
g++ -DHAVE_CONFIG_H -I. -I..    -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11 -MT utils/snap_collector_dpdkstats-common.o -MD -MP -MF utils/.deps/snap_collector_dpdkstats-common.Tpo -c -o utils/snap_collector_dpdkstats-common.o `test -f 'utils/common.cc' || echo './'`utils/common.cc
mv -f utils/.deps/snap_collector_dpdkstats-common.Tpo utils/.deps/snap_collector_dpdkstats-common.Po
g++ -I/usr/include/dpdk -Wall -pthread -I/usr/local/include -I/usr/local/include -I./utils -g -O2 -std=gnu++11   -o snap-collector-dpdkstats snap_collector_dpdkstats-dpdkstats.o snap_collector_dpdkstats-dpdkstats_collector.o utils/snap_collector_dpdkstats-dpdk.o utils/snap_collector_dpdkstats-log.o utils/snap_collector_dpdkstats-common.o  -lsnap -L/usr/local/lib -lprotobuf -pthread -lpthread -L/usr/local/lib -lgrpc++ -L/usr/lib -Wl,--as-needed -Wl,-ldpdk
make[2]: Leaving directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro/src'
make[2]: Entering directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro'
make[2]: Leaving directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro'
make[1]: Leaving directory '/root/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro'


root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ldd src/snap-collector-dpdkstats
        linux-vdso.so.1 =>  (0x00007ffff7ffd000)
        libsnap.so.0 => /usr/local/lib/libsnap.so.0 (0x00007ffff7c5e000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffff7a23000)
        libgrpc++.so.1 => /usr/local/lib/libgrpc++.so.1 (0x00007ffff779d000)
        libethdev.so.4.1 => /usr/lib/libethdev.so.4.1 (0x00007ffff750f000)
        librte_eal.so.2.1 => /usr/lib/librte_eal.so.2.1 (0x00007ffff72b1000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffff6f2e000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffff6c25000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffff6a0f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffff6645000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ffff642b000)
        libgrpc.so.1 => /usr/local/lib/libgrpc.so.1 (0x00007ffff617c000)
        librte_mempool.so.2.1 => /usr/lib/librte_mempool.so.2.1 (0x00007ffff5f74000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffff5d70000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffff5b68000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007ffff58fe000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007ffff54ba000)
        librte_ring.so.1.1 => /usr/lib/librte_ring.so.1.1 (0x00007ffff52b7000)


##=========SUCCESS with explicitly set --base-virtaddr ===============================================

## in testpmd terminal window
root at silpixa00393941:~/kmatczax# $RTE_SDK/tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:06:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' drv=igb_uio unused=i40e

Network devices using kernel driver
===================================
0000:03:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=eth0 drv=ixgbe unused=igb_uio *Active*
0000:03:00.1 'Ethernet Controller 10-Gigabit X540-AT2' if=eth1 drv=ixgbe unused=igb_uio

Other network devices
=====================
<none>

root at silpixa00393941:~/kmatczax# $RTE_SDK/build/app/testpmd -c 0xf -n 2 -v --base-virtaddr=0x2aaa4a90000 --socket-mem 1024,0 -- --burst=64
EAL: Detected 28 lcore(s)
EAL: RTE Version: 'DPDK 16.07.0'
EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
EAL: Probing VFIO support...
PMD: bnxt_rte_pmd_init() called for (null)
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1584 rte_i40e_pmd
PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001dc9
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0

Warning! Cannot handle an odd number of ports with the current port topology. Configuration must be changed to have an even number of ports, or relaunch application with --port-topology=chained

Configuring Port 0 (socket 0)
Port 0: 68:05:CA:2D:41:C0
Checking link statuses...
Port 0 Link Down
Done
No commandline core given, start packet forwarding

Warning! Cannot handle an odd number of ports with the current port topology. Configuration must be changed to have an even number of ports, or relaunch application with --port-topology=chained

io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support disabled, MP over anonymous pages disabled
Logical Core 1 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  io packet forwarding - CRC stripping disabled - packets/burst=64
  nb forwarding cores=1 - nb forwarding ports=1
  RX queues=1 - RX desc=128 - RX free threshold=32
  RX threshold registers: pthresh=8 hthresh=8 wthresh=0
  TX queues=1 - TX desc=512 - TX free threshold=32
  TX threshold registers: pthresh=32 hthresh=0 wthresh=0
  TX RS bit threshold=32 - TXQ flags=0xf01
Press enter to exit


## switched to other terminal window for running secondary process 
root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ./src/snap-collector-dpdkstats
EAL: Detected 28 lcore(s)
EAL: Probing VFIO support...
PMD: bnxt_rte_pmd_init() called for (null)
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1584 rte_i40e_pmd

## EAL params of secondary process 
Nov 16 11:17:21 silpixa00393941 dpdkstats[68951]: EAL init params: snap-collector-dpdkstats -c 0xf0 -n 2 --base-virtaddr 0x2aaa4a90000 --proc-type secondary
Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: PMD: bnxt_rte_pmd_init() called for (null)
Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL: PCI device 0000:03:00.0 on NUMA socket 0
Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL: PCI device 0000:03:00.1 on NUMA socket 0
Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL: PCI device 0000:06:00.0 on NUMA socket 0
Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: EAL:   probe driver: 8086:1584 rte_i40e_pmd
Nov 16 11:17:21 silpixa00393941 snap-collector-dpdkstats[68951]: snap-collector-dpdkstats:dpdk_helper_eal_init:291 DPDK_HELPER_INITIALIZING_EAL (done)


## ========= FAILURE  with no  --base-virtaddr =============================================
## in testpmd terminal window

root at silpixa00393941:~/kmatczax# $RTE_SDK/build/app/testpmd -c 0xf -n 2 -v --socket-mem 1024,0
-- --burst=64
EAL: Detected 28 lcore(s)
EAL: RTE Version: 'DPDK 16.07.0'
EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
EAL: Probing VFIO support...
PMD: bnxt_rte_pmd_init() called for (null)
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1584 rte_i40e_pmd
PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001dc9
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0

Warning! Cannot handle an odd number of ports with the current port topology. Configuration must be changed to have an even number of ports, or relaunch application with --port-topology=chained

Configuring Port 0 (socket 0)
Port 0: 68:05:CA:2D:41:C0
Checking link statuses...
Port 0 Link Down
Done
No commandline core given, start packet forwarding

Warning! Cannot handle an odd number of ports with the current port topology. Configuration must be changed to have an even number of ports, or relaunch application with --port-topology=chained

io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support disabled, MP over anonymous pages disabled
Logical Core 1 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  io packet forwarding - CRC stripping disabled - packets/burst=64
  nb forwarding cores=1 - nb forwarding ports=1
  RX queues=1 - RX desc=128 - RX free threshold=32
  RX threshold registers: pthresh=8 hthresh=8 wthresh=0
  TX queues=1 - TX desc=512 - TX free threshold=32
  TX threshold registers: pthresh=32 hthresh=0 wthresh=0
  TX RS bit threshold=32 - TXQ flags=0xf01
Press enter to exit

## secondary process window 
root at silpixa00393941:~/kmatczax/tmp/dpdk_xstats-snap-plugins_PR_repro# ./src/snap-collector-dpdkstats
EAL: Detected 28 lcore(s)
PANIC in rte_eal_config_reattach():
Cannot mmap memory for rte_config
8: [./src/snap-collector-dpdkstats() [0x408389]]
7: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ffff6464830]]
6: [./src/snap-collector-dpdkstats() [0x407f5c]]
5: [./src/snap-collector-dpdkstats() [0x4088db]]
4: [./src/snap-collector-dpdkstats() [0x418e49]]
3: [/usr/lib/librte_eal.so.2.1(rte_eal_init+0xe83) [0x7ffff70b4e83]]
2: [/usr/lib/librte_eal.so.2.1(__rte_panic+0xcb) [0x7ffff70b3a25]]
1: [/usr/lib/librte_eal.so.2.1(rte_dump_stack+0x2b) [0x7ffff70bc60b]]
Aborted (core dumped)

##EAL params of secondary process
Nov 16 11:21:38 silpixa00393941 dpdkstats[68968]: EAL init params: snap-collector-dpdkstats -c 0xf0 -n 2 --proc-type secondary


Thanks in advance for help
Krzysztof

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
Previous message: [dpdk-users] vm2vm and host2vm performance using dpdk 16.11
Next message: [dpdk-users] PANIC in rte_eal_config_reattach() with C++ app
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the users mailing list