[dpdk-dev] SR-IOV/DPDK/VPP with vfio-pci

Chandrasekar Kannan ckannan at console.to
Wed Jun 1 20:21:50 CEST 2016
Previous message: [dpdk-dev] [PATCH v7 5/5] mbuf: allow apps to change default mempool ops
Next message: [dpdk-dev] SR-IOV/DPDK/VPP with vfio-pci
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
I have been attempting to VPP (dpdk) application with SR-IOV enabled.
Using the vfio-pci driver/i40e/XL710nics/.
I'm encountering a SEGV on the i40e code path.

Has anyone else seen this behaviour ?.

full mail thread discussion with vpp-dev is here -
https://lists.fd.io/pipermail/vpp-dev/2016-June/001348.html


--------------------------------
here's a backtrace during vpp crash...

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f2b811fd700 (LWP 8725)]
0x0000000000455539 in rx_recv_pkts ()
(gdb) backtrace
#0  0x0000000000455539 in rx_recv_pkts ()
#1  0x0000000000455e76 in i40e_recv_pkts_bulk_alloc ()
#2  0x00007f2be43d8bf7 in rte_eth_rx_burst (nb_pkts=256,
rx_pkts=0x7f2bb1e66a80, queue_id=1, port_id=<optimized out>)
    at
/w/workspace/vpp-merge-master-centos7/build-root/install-vpp-native/dpdk/include/rte_ethdev.h:2641
#3  dpdk_rx_burst (dm=0x7f2be4673980 <dpdk_main>, queue_id=1,
xd=0x7f2bb1898540)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/dpdk_priv.h:64
#4  dpdk_device_input (dm=0x7f2be4673980 <dpdk_main>, queue_id=1,
cpu_index=<optimized out>, node=0x7f2bb1ef59fc, xd=<optimized out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:508
#5  dpdk_input_rss (vm=<optimized out>, node=0x7f2bb1ef59fc, f=<optimized
out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:821
#6  0x00007f2be47eb77a in dispatch_node (vm=vm at entry=0x7f2bb1f02cd4,
node=node at entry=0x7f2bb1ef59fc, type=type at entry=VLIB_NODE_TYPE_INPUT,
    dispatch_state=dispatch_state at entry=VLIB_NODE_STATE_POLLING,
frame=frame at entry=0x0, last_time_stamp=1482097661494035)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:996
#7  0x00007f2be43db6d8 in dpdk_worker_thread_internal (have_io_threads=0,
callback=0x0, vm=0x7f2bb1f02cd4)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:209
#8  dpdk_worker_thread (w=<optimized out>, io_name=<optimized out>,
callback=0x0)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:265
#9  0x00007f2be3791530 in clib_calljmp () at
/w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110
#10 0x00007f2b811fcc40 in ?? ()
#11 0x00000000004eb9c7 in eal_thread_loop ()
---Type <return> to continue, or q <return> to quit---
#12 0x0000000000000000 in ?? ()
(gdb)
(gdb) thread apply all bt

Thread 10 (Thread 0x7f2b8c685700 (LWP 8722)):
#0  0x00007f2be32584ad in accept () at ../sysdeps/unix/syscall-template.S:81
#1  0x00000000004ef178 in pci_vfio_mp_sync_thread ()
#2  0x00007f2be3251dc5 in start_thread (arg=0x7f2b8c685700) at
pthread_create.c:308
#3  0x00007f2be2992ced in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 9 (Thread 0x7f2b821ff700 (LWP 8723)):
#0  0x00007f2be29932c3 in epoll_wait () at
../sysdeps/unix/syscall-template.S:81
#1  0x00000000004efe04 in eal_intr_thread_main ()
#2  0x00007f2be3251dc5 in start_thread (arg=0x7f2b821ff700) at
pthread_create.c:308
#3  0x00007f2be2992ced in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 8 (Thread 0x7f2b819fe700 (LWP 8724)):
#0  0x000000000045526b in rx_recv_pkts ()
#1  0x0000000000455e76 in i40e_recv_pkts_bulk_alloc ()
#2  0x00007f2be43d8bf7 in rte_eth_rx_burst (nb_pkts=256,
rx_pkts=0x7f2bb1e75580, queue_id=0, port_id=<optimized out>)
    at
/w/workspace/vpp-merge-master-centos7/build-root/install-vpp-native/dpdk/include/rte_ethdev.h:2641
#3  dpdk_rx_burst (dm=0x7f2be4673980 <dpdk_main>, queue_id=0,
xd=0x7f2bb1a1fd40)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/dpdk_priv.h:64
---Type <return> to continue, or q <return> to quit---
#4  dpdk_device_input (dm=0x7f2be4673980 <dpdk_main>, queue_id=0,
cpu_index=<optimized out>, node=0x7f2bb1ea18cc, xd=<optimized out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:508
#5  dpdk_input_rss (vm=<optimized out>, node=0x7f2bb1ea18cc, f=<optimized
out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:821
#6  0x00007f2be47eb77a in dispatch_node (vm=vm at entry=0x7f2bb1f237d4,
node=node at entry=0x7f2bb1ea18cc, type=type at entry=VLIB_NODE_TYPE_INPUT,
    dispatch_state=dispatch_state at entry=VLIB_NODE_STATE_POLLING,
frame=frame at entry=0x0, last_time_stamp=1482097666654800)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:996
#7  0x00007f2be43db6d8 in dpdk_worker_thread_internal (have_io_threads=0,
callback=0x0, vm=0x7f2bb1f237d4)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:209
#8  dpdk_worker_thread (w=<optimized out>, io_name=<optimized out>,
callback=0x0)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:265
#9  0x00007f2be3791530 in clib_calljmp () at
/w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110
#10 0x00007f2b819fdc40 in ?? ()
#11 0x00000000004eb9c7 in eal_thread_loop ()
#12 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f2b811fd700 (LWP 8725)):
#0  0x0000000000455539 in rx_recv_pkts ()
#1  0x0000000000455e76 in i40e_recv_pkts_bulk_alloc ()
#2  0x00007f2be43d8bf7 in rte_eth_rx_burst (nb_pkts=256,
rx_pkts=0x7f2bb1e66a80, queue_id=1, port_id=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at
/w/workspace/vpp-merge-master-centos7/build-root/install-vpp-native/dpdk/include/rte_ethdev.h:2641
#3  dpdk_rx_burst (dm=0x7f2be4673980 <dpdk_main>, queue_id=1,
xd=0x7f2bb1898540)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/dpdk_priv.h:64
#4  dpdk_device_input (dm=0x7f2be4673980 <dpdk_main>, queue_id=1,
cpu_index=<optimized out>, node=0x7f2bb1ef59fc, xd=<optimized out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:508
#5  dpdk_input_rss (vm=<optimized out>, node=0x7f2bb1ef59fc, f=<optimized
out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:821
#6  0x00007f2be47eb77a in dispatch_node (vm=vm at entry=0x7f2bb1f02cd4,
node=node at entry=0x7f2bb1ef59fc, type=type at entry=VLIB_NODE_TYPE_INPUT,
    dispatch_state=dispatch_state at entry=VLIB_NODE_STATE_POLLING,
frame=frame at entry=0x0, last_time_stamp=1482097661494035)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:996
#7  0x00007f2be43db6d8 in dpdk_worker_thread_internal (have_io_threads=0,
callback=0x0, vm=0x7f2bb1f02cd4)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:209
#8  dpdk_worker_thread (w=<optimized out>, io_name=<optimized out>,
callback=0x0)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:265
#9  0x00007f2be3791530 in clib_calljmp () at
/w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110
#10 0x00007f2b811fcc40 in ?? ()
#11 0x00000000004eb9c7 in eal_thread_loop ()
#12 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f2b809fc700 (LWP 8726)):
---Type <return> to continue, or q <return> to quit---
#0  0x000000000045539e in rx_recv_pkts ()
#1  0x0000000000455e76 in i40e_recv_pkts_bulk_alloc ()
#2  0x00007f2be43d8bf7 in rte_eth_rx_burst (nb_pkts=256,
rx_pkts=0x7f2bb1da8a00, queue_id=2, port_id=<optimized out>)
    at
/w/workspace/vpp-merge-master-centos7/build-root/install-vpp-native/dpdk/include/rte_ethdev.h:2641
#3  dpdk_rx_burst (dm=0x7f2be4673980 <dpdk_main>, queue_id=2,
xd=0x7f2bb1898540)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/dpdk_priv.h:64
#4  dpdk_device_input (dm=0x7f2be4673980 <dpdk_main>, queue_id=2,
cpu_index=<optimized out>, node=0x7f2bb1ebc7f8, xd=<optimized out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:508
#5  dpdk_input_rss (vm=<optimized out>, node=0x7f2bb1ebc7f8, f=<optimized
out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/node.c:821
#6  0x00007f2be47eb77a in dispatch_node (vm=vm at entry=0x7f2bb1ee29d4,
node=node at entry=0x7f2bb1ebc7f8, type=type at entry=VLIB_NODE_TYPE_INPUT,
    dispatch_state=dispatch_state at entry=VLIB_NODE_STATE_POLLING,
frame=frame at entry=0x0, last_time_stamp=1482097658614609)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:996
#7  0x00007f2be43db6d8 in dpdk_worker_thread_internal (have_io_threads=0,
callback=0x0, vm=0x7f2bb1ee29d4)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:209
#8  dpdk_worker_thread (w=<optimized out>, io_name=<optimized out>,
callback=0x0)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:265
#9  0x00007f2be3791530 in clib_calljmp () at
/w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110
#10 0x00007f2b809fbc40 in ?? ()
#11 0x00000000004eb9c7 in eal_thread_loop ()
---Type <return> to continue, or q <return> to quit---
#12 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f2b801fb700 (LWP 8727)):
#0  0x00007f2be47eb728 in dispatch_node (vm=vm at entry=0x7f2bb1ec2614,
node=node at entry=0x7f2bb1da5e60, type=type at entry=VLIB_NODE_TYPE_INPUT,
    dispatch_state=dispatch_state at entry=VLIB_NODE_STATE_POLLING,
frame=frame at entry=0x0, last_time_stamp=1482097666665442)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:924
#1  0x00007f2be43db6d8 in dpdk_worker_thread_internal (have_io_threads=0,
callback=0x0, vm=0x7f2bb1ec2614)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:209
#2  dpdk_worker_thread (w=<optimized out>, io_name=<optimized out>,
callback=0x0)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:265
#3  0x00007f2be3791530 in clib_calljmp () at
/w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110
#4  0x00007f2b801fac40 in ?? ()
#5  0x00000000004eb9c7 in eal_thread_loop ()
#6  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f2b7f9fa700 (LWP 8728)):
#0  0x00007f2be43db6cc in dpdk_worker_thread_internal (have_io_threads=0,
callback=0x0, vm=0x7f2bb1ea2294)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:209
#1  dpdk_worker_thread (w=<optimized out>, io_name=<optimized out>,
callback=0x0)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:265
---Type <return> to continue, or q <return> to quit---
#2  0x00007f2be3791530 in clib_calljmp () at
/w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110
#3  0x00007f2b7f9f9c40 in ?? ()
#4  0x00000000004eb9c7 in eal_thread_loop ()
#5  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f2b7f1f9700 (LWP 8729)):
#0  vlib_node_runtime_update_stats (n_clocks=205, n_vectors=0, n_calls=1,
node=0x7f2bb1d782d0, vm=0x7f2bb1e81f94)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:629
#1  dispatch_node (vm=vm at entry=0x7f2bb1e81f94, node=node at entry=0x7f2bb1d782d0,
type=type at entry=VLIB_NODE_TYPE_INPUT,
    dispatch_state=dispatch_state at entry=VLIB_NODE_STATE_POLLING,
frame=frame at entry=0x0, last_time_stamp=1482097666680456)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:1005
#2  0x00007f2be43db6d8 in dpdk_worker_thread_internal (have_io_threads=0,
callback=0x0, vm=0x7f2bb1e81f94)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:209
#3  dpdk_worker_thread (w=<optimized out>, io_name=<optimized out>,
callback=0x0)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vnet/vnet/devices/dpdk/threads.c:265
#4  0x00007f2be3791530 in clib_calljmp () at
/w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110
#5  0x00007f2b7f1f8c40 in ?? ()
#6  0x00000000004eb9c7 in eal_thread_loop ()
#7  0x0000000000000000 in ?? ()

---Type <return> to continue, or q <return> to quit---
Thread 2 (Thread 0x7f2b7e9f8700 (LWP 8730)):
#0  0x00007f2be325896d in nanosleep () at
../sysdeps/unix/syscall-template.S:81
#1  0x0000000000413916 in ip46_fib_stats_delay.isra.6 ()
#2  0x00000000004190b6 in stats_thread_fn ()
#3  0x00007f2be3791530 in clib_calljmp () at
/w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110
#4  0x00007f2b7e9f7da0 in ?? ()
#5  0x00007f2be3251dc5 in start_thread (arg=0x0) at pthread_create.c:308
#6  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f2be52898c0 (LWP 8721)):
#0  0x00007f2be2992f27 in __GI_epoll_pwait (epfd=5, events=0x7f2bb2155424,
maxevents=256, timeout=timeout at entry=10,
    set=set at entry=0x7f2be4c522e0 <unblock_all_signals.16333>) at
../sysdeps/unix/sysv/linux/epoll_pwait.c:48
#1  0x00007f2be4a429fb in linux_epoll_input (vm=0x7e13c0
<vlib_global_main>, node=<optimized out>, frame=<optimized out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/unix/input.c:146
#2  0x00007f2be47eb77a in dispatch_node (vm=0x7e13c0 <vlib_global_main>,
node=0x7f2bb2254cc0, type=<optimized out>, dispatch_state=<optimized out>,
    frame=<optimized out>, last_time_stamp=1482097636972836) at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:996
#3  0x00007f2be47ebea1 in vlib_main_loop (vm=0x7e13c0 <vlib_global_main>)
at /w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:1403
#4  vlib_main (vm=vm at entry=0x7e13c0 <vlib_global_main>, input=input at entry
=0x7f2bb2786fa0)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/main.c:1615
#5  0x00007f2be4a435f3 in thread0 (arg=8262592) at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/unix/main.c:425
---Type <return> to continue, or q <return> to quit---
#6  0x00007f2be3791530 in clib_calljmp () at
/w/workspace/vpp-merge-master-centos7/build-data/../vppinfra/vppinfra/longjmp.S:110
#7  0x00007ffc98a68d60 in ?? ()
#8  0x00007f2be4a43e31 in vlib_unix_main (argc=<optimized out>,
argv=<optimized out>)
    at
/w/workspace/vpp-merge-master-centos7/build-data/../vlib/vlib/unix/main.c:485
#9  0x0000000000000000 in ?? ()
(gdb)
(gdb)
(gdb)
(gdb)
-------------------------------------------------------------------
Previous message: [dpdk-dev] [PATCH v7 5/5] mbuf: allow apps to change default mempool ops
Next message: [dpdk-dev] SR-IOV/DPDK/VPP with vfio-pci
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the dev mailing list