[dpdk-dev] Testpmd with SR-IOV.

Suresh Vinapamula suresh.vin at gmail.com
Mon Dec 1 20:22:18 CET 2014


Hi

I am trying to run latest DPDK on SR-IOV interface in VM environment.
Interfaces are detected well in VM environment.

When I started traffic through testpmd, it indicates both tx and rx are
zero. I tried debugging. Wondering if you could point me some steps which I
might be missing? Thanks in advance.

Here are the steps that I followed.

SR-IOV interfaces are detected in VM environment.

Lspci indicate below interfaces in VM environment and ethtool indicates
links are up and link speeds are fine.

00:08.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
Virtual Function (rev 01)
00:09.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
Virtual Function (rev 01)

root at regress-KVM:/dev/abc/dpdk/tools# ethtool eth2
Settings for eth2:
Supported ports: [ ]
Supported link modes:   10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes:  Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: Other
PHYAD: 0
Transceiver: Unknown!
Auto-negotiation: off
Current message level: 0x00000007 (7)
      drv probe link
Link detected: yes
root at regress-KVM:/dev/abc/dpdk/tools# ethtool eth3
Settings for eth3:
Supported ports: [ ]
Supported link modes:   10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes:  Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: Other
PHYAD: 0
Transceiver: Unknown!
Auto-negotiation: off
Current message level: 0x00000007 (7)
      drv probe link
Link detected: yes


Loaded uio module.

root at regress-KVM:/dev/abc/dpdk/tools# modprobe uio
root at regress-KVM:/dev/abc/dpdk/tools# insmod
../x86_64-native-linuxapp-gcc/kmod/igb_uio.ko



Bound interfaces to igb_uio.

root at regress-KVM:/dev/abc/dpdk/tools# python dpdk_nic_bind.py
--bind=igb_uio 00:09.0
root at regress-KVM:/dev/abc/dpdk/tools# python dpdk_nic_bind.py
--bind=igb_uio 00:08.0


Verified that interfaces are bound to igb_uio.

root at regress-KVM:/dev/abc/dpdk/tools# python dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:00:08.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio
unused=vfio-pci
0000:00:09.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio
unused=vfio-pci

Network devices using kernel driver
===================================
0000:00:03.0 'Virtio network device' if= drv=virtio-pci
unused=igb_uio,vfio-pci
0000:00:07.0 '82599 Ethernet Controller Virtual Function' if=eth1
drv=ixgbevf unused=igb_uio,vfio-pci

Other network devices
=====================
<none>
root at regress-KVM:/dev/abc/dpdk/tools#


Built test-pmd and executed and links detected.

root at regress-KVM:/dev/abc/dpdk/app/test-pmd# ./testpmd -c 0xf -n 4  -- -i
--nb-cores=2
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Support maximum 64 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL:   unsupported IOMMU type!
EAL: VFIO support could not be initialized
EAL: Setting up memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd8e1200000 (size = 0x200000)
EAL: Ask a virtual area of 0x7f000000 bytes
EAL: Virtual area found at 0x7fd862000000 (size = 0x7f000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd861c00000 (size = 0x200000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7fd861600000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd861200000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd860e00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd860a00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd860600000 (size = 0x200000)
EAL: Requesting 1024 pages of size 2MB from socket 0
EAL: TSC frequency is ~1999999 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable
clock cycles !
EAL: Master core 0 is ready (tid=e281a840)
EAL: Core 3 is ready (tid=5e9f0700)
EAL: Core 2 is ready (tid=5f1f1700)
EAL: Core 1 is ready (tid=5f9f2700)
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   0000:00:03.0 not managed by UIO driver, skipping
EAL: PCI device 0000:00:07.0 on NUMA socket -1
EAL:   probe driver: 8086:10ed rte_ixgbevf_pmd
EAL:   0000:00:07.0 not managed by UIO driver, skipping
EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL:   probe driver: 8086:10ed rte_ixgbevf_pmd
EAL:   PCI memory mapped at 0x7fd8e2828000
EAL:   PCI memory mapped at 0x7fd8e2824000
EAL: PCI device 0000:00:09.0 on NUMA socket -1
EAL:   probe driver: 8086:10ed rte_ixgbevf_pmd
EAL:   PCI memory mapped at 0x7fd8e2820000
EAL:   PCI memory mapped at 0x7fd8e27e0000
Interactive-mode selected
Configuring Port 0 (socket 0)
Port 0: 52:54:00:E2:90:42
Configuring Port 1 (socket 0)
Port 1: 52:54:00:CA:0D:17
Checking link statuses...
Port 0 Link Up - speed 10000 Mbps - full-duplex
Port 1 Link Up - speed 10000 Mbps - full-duplex
Done
testpmd>

When I start and stop traffic, stats show that zero traffic xmitted.


testpmd>  start
  io packet forwarding - CRC stripping disabled - packets/burst=32
  nb forwarding cores=2 - nb forwarding ports=2
  RX queues=1 - RX desc=128 - RX free threshold=0
  RX threshold registers: pthresh=8 hthresh=8 wthresh=0
  TX queues=1 - TX desc=512 - TX free threshold=0
  TX threshold registers: pthresh=32 hthresh=0 wthresh=0
  TX RS bit threshold=0 - TXQ flags=0x0
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0
----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0

----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1
----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0

----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all
ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.
testpmd>


I debugged this to some extent and below are my observations. For some
reason I am not able to compile old dpdk code from the website. Can you
point me to your code?

When I start traffic in testpmd, I see two thread taking 100%cpu
utilization. Ideally I would expect one thread to be rx and another to be
tx. One thread does the tx and while other thread does rx, is my
understanding correct?

But what I see is both the treads are in rx path!? Is that expected? Am I
missing something? I believe you must have automated these steps in scripts
for vmx. Can you please point me to them?

 PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+
COMMAND

 8496 root      20   0 2291m 5844 5532 R  99.9  0.0   2:27.13
testpmd

 8495 root      20   0 2291m 5844 5532 R  99.8  0.0   2:27.12
testpmd

 8896 root      20   0 24972 1712 1124 R   0.3  0.0   0:05.60
top

    1 root      20   0 27352 3200 1416 S   0.0  0.0   0:01.42 init



(gdb) info threads
  Id   Target Id         Frame
  5    Thread 0x7f7d029f3700 (LWP 8494) "testpmd" 0x00007f7d84234083 in
epoll_wait () at ../sysdeps/unix/syscall-template.S:81
* 4    Thread 0x7f7d021f2700 (LWP 8495) "testpmd" rte_eth_rx_burst
(port_id=0 '\000', queue_id=0, rx_pkts=0x7f7d021f0d10, nb_pkts=32) at
/dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304
  3    Thread 0x7f7d019f1700 (LWP 8496) "testpmd" rte_eth_rx_burst
(port_id=2 '\002', queue_id=0, rx_pkts=0x7f7d019efd10, nb_pkts=32) at
/dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304
  2    Thread 0x7f7d011f0700 (LWP 8497) "testpmd" 0x00007f7d8450fc9d in
read () at ../sysdeps/unix/syscall-template.S:81
  1    Thread 0x7f7d85037840 (LWP 8493) "testpmd" 0x00007f7d8450fc9d in
read () at ../sysdeps/unix/syscall-template.S:81
(gdb) thread 4
[Switching to thread 4 (Thread 0x7f7d021f2700 (LWP 8495))]
#0  rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f7d021f0d10,
nb_pkts=32) at
/dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304
2304}
(gdb) bt
#0  rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f7d021f0d10,
nb_pkts=32) at
/dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304
#1  0x0000000000438464 in pkt_burst_io_forward (fs=0x7f7d052f6200) at
iofwd.c:97
#2  0x0000000000427596 in run_pkt_fwd_on_lcore (fc=0x7f7d052f7dc0,
pkt_fwd=0x43841f <pkt_burst_io_forward>) at testpmd.c:952
#3  0x00000000004275df in start_pkt_forward_on_core
(fwd_arg=0x7f7d052f7dc0) at testpmd.c:959
#4  0x0000000000498759 in eal_thread_loop (arg=<optimized out>) at
/dev/abc/dpdk/lib/librte_eal/linuxapp/eal/eal_thread.c:224
#5  0x00007f7d84508f6e in start_thread (arg=0x7f7d021f2700) at
pthread_create.c:311
#6  0x00007f7d842339cd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f7d019f1700 (LWP 8496))]
#0  rte_eth_rx_burst (port_id=2 '\002', queue_id=0, rx_pkts=0x7f7d019efd10,
nb_pkts=32) at
/dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304
2304}
(gdb) bt
#0  rte_eth_rx_burst (port_id=2 '\002', queue_id=0, rx_pkts=0x7f7d019efd10,
nb_pkts=32) at
/dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304
#1  0x0000000000438464 in pkt_burst_io_forward (fs=0x7f7d052f6100) at
iofwd.c:97
#2  0x0000000000427596 in run_pkt_fwd_on_lcore (fc=0x7f7d052f7d40,
pkt_fwd=0x43841f <pkt_burst_io_forward>) at testpmd.c:952
#3  0x00000000004275df in start_pkt_forward_on_core
(fwd_arg=0x7f7d052f7d40) at testpmd.c:959
#4  0x0000000000498759 in eal_thread_loop (arg=<optimized out>) at
/dev/abc/dpdk/lib/librte_eal/linuxapp/eal/eal_thread.c:224
#5  0x00007f7d84508f6e in start_thread (arg=0x7f7d019f1700) at
pthread_create.c:311
#6  0x00007f7d842339cd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb)


More information about the dev mailing list