[dpdk-dev] ixgbevf does not recover from pf reset

[David Marchand]

Hello Helin, Konstantin,

I noticed this issue quite some time ago (maybe around dpdk-1.3.0, not
sure) but had no time to investigate/report.
I hit it again with dpdk-2.2.0-rc2, so maybe all dpdk versions are impacted.

The test is quite simple :
- send icmp packets from a system (tn) to itself using two 10G interfaces
connected to two VF ports of a dual port 82599 nic,
- run testpmd in a vm owning those two VF ports,
- reset a pf interface in the host,
- then no connectivity on the associated VF port unless it is completely
restarted.

In my previous test, I think I also unplugged the fiber then replugged
(without touching the pf interface status on the host) and observed the
same issue.

I looked at ixgbevf pmd, but I can't find a place where the PF asking for
reset would be handled except at devinit.



Here is the full description of my test :

The system owning the two VF ports is a ubuntu 14.04 virtual machine
(3.13.0-71-generic kernel).
The host running this virtual machine is running ubuntu 14.04 as well (but
kernel is newer 3.19.0-39-generic).


- host setup :
echo 3072
>/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 3072
>/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
mkdir -p
/mnt/huge
mount -t hugetlbfs none /mnt/huge

echo 1 >
"/sys/bus/pci/devices/0000:83:00.0/sriov_numvfs"
bridge link set dev ixgbe0 hwmode veb
ip link set dev ixgbe0 vf 0 mac 00:09:c0:0e:4e:64
ip link set dev ixgbe0 vf 0 spoofchk off
ip link set dev ixgbe0 mtu 9000
ip link set dev ixgbe0 up
ip link set dev ixgbe0 address 00:09:c0:0e:4e:64

echo 1 >
"/sys/bus/pci/devices/0000:83:00.1/sriov_numvfs"
bridge link set dev ixgbe1 hwmode veb
ip link set dev ixgbe1 vf 0 mac 00:09:c0:0e:4e:65
ip link set dev ixgbe1 vf 0 spoofchk off
ip link set dev ixgbe1 mtu 9000
ip link set dev ixgbe1 up
ip link set dev ixgbe1 address 00:09:c0:0e:4e:65

qemu-system-x86_64 -k fr -daemonize -S --enable-kvm -m 3G -mem-path
/mnt/huge -cpu host -smp cores=6,threads=2,sockets=1 -serial
telnet::53714,server,nowait -serial null -monitor
telnet::36576,server,nowait -device
virtio-net,mac=DE:AD:DE:01:02:03,netdev=user.0,addr=03 -netdev
user,id=user.0 -device pci-assign,host=0000:83:10.0,addr=04 -device
pci-assign,host=0000:83:10.1,addr=05 -hda ubuntu-14.04-template.qcow2
-snapshot -vga none -display none


- guest setup :
echo 1024 >
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
mkdir -p
/mnt/huge
mount -t hugetlbfs none /mnt/huge

modprobe uio
modprobe igb_uio
echo 0000:00:04.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind
echo 0000:00:05.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind
echo 8086 10ed > /sys/bus/pci/drivers/igb_uio/new_id

testpmd -w0000:00:04.0 -w0000:00:05.0 --huge-dir=/mnt/huge -n 4 -c 0xe -- -i
[snip]
Port 0:
00:09:C0:0E:4E:64
[snip]
Port 1:
00:09:C0:0E:4E:65
Checking link
statuses...
Port 0 Link Up - speed 10000 Mbps -
full-duplex
Port 1 Link Up - speed 10000 Mbps -
full-duplex
Done

testpmd>
start


- tn setup :
ip link set dev xaui0 up
ip link set dev xaui1 up
ip addr add 1.1.1.1/24 dev xaui0
arp -i xaui0 -s 1.1.1.2 00:09:c0:0e:4e:64



- From here, on tn, send icmp packets with a tcpdump running background on
the other iface :
ping 1.1.1.2
PING 1.1.1.2 (1.1.1.2): 56 data
bytes
03:10:15.003047 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
(0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 1
03:10:16.003151 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
(0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 2
03:10:17.003246 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
(0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 3


- Ok, now, kick ixgbe0 pf interface down / up on host system.
ip link set dev ixgbe0 down
ip link set dev ixgbe0 up


- On tn, no packet is received anymore, the only workaround I found is to
stop and restart the port 0 :

testpmd>
stop
[snip]
Done.

testpmd> port stop
all
Stopping
ports...
Checking link
statuses...
Port 0 Link
Down
Port 1 Link Up - speed 10000 Mbps -
full-duplex
Done


Funny thing here, the link is reported down, while the pf is already back
up.

testpmd> port start all
PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
size no less than 4 (port=0).
Port 0:
00:09:C0:0E:4E:64
PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
size no less than 4 (port=1).
Port 1:
00:09:C0:0E:4E:65
Checking link
statuses...
Port 0 Link Up - speed 10000 Mbps -
full-duplex
Port 1 Link Up - speed 10000 Mbps -
full-duplex
Done



Thanks.

-- 
David Marchand