[dpdk-dev] After a port is bound to dpdk, when 'ip addr' or 'ip link add ~vlan' command against non-dpdk port , Linux kernel hangs

[Joo Kim]

This issue(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1570195
)seems to be some similar  to what I am facing.
So, I tried its fix patches
(
https://github.com/openstack/fuel-plugin-ovs/blob/master/ovs_build/dpdk_2.2.0/debian/patches/ubuntu-backport-36-pci-identify-devices-not-managed-by-any-kernel-drive.patch
https://github.com/openstack/fuel-plugin-ovs/blob/master/ovs_build/dpdk_2.2.0/debian/patches/ubuntu-backport-37-pci-ignore-devices-already-managed-in-Linux-when-map.patch
)
But, it didnot help.

BTW,  what I meant by 'Linux kernel hangs' is soft lockup.
(I see following msg in dmesg
[252460.127003] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322]
[252488.127003] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322]
[252493.698003] INFO: rcu_sched self-detected stall on CPU { 1}  (t=60001
jiffies g=9321477 c=9321476 q=0)
[252493.699003] INFO: rcu_sched detected stalls on CPUs/tasks:
[252520.127002] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322]
[252548.127003] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322]

)

On Fri, Dec 9, 2016 at 3:27 PM, Joo Kim <itsolution at gmail.com> wrote:

> Hello,
> I am using dpdk 2.2.0 in this VM where several NIC ports are available.
>
> Also, uio, kni drivers are installed.
> [root at mylinux]# lsmod | grep uio
> igb_uio                13224  0
> uio                    19259  1 igb_uio
> [root at mylinux]# lsmod | grep kni
> rte_kni               294375  0
> i2c_core               40325  2 i2c_dev,rte_kni
> [root at mylinux]# uname -r
> 3.10.0-229.20.1.el7.x86_64
>
> Weirdly,  once I bind a network port to dpdk,   after that, when I run 'ip
> addr' or 'ip link add ~vlan' command against non-dpdk port ,  this VM's
> Linux kernel hangs.
>
> For example, when I run  'ip link add ~vlan' command, then  following
> dmesg is seen and Linux hangs.
>
> Any idea on root cause/how to fix?
>
> {noformat}
> [ 4884.158003]  virtio dm_mirror dm_region_hash dm_log dm_mod loop [last
> unloaded: openvswitch]
> [ 4884.158003] CPU: 3 PID: 15060 Comm: dpdk_nic_bind.p Tainted: GF
> O--------------   3.10.0-229.20.1.el7.x86_64 #1
> [ 4884.158003] Hardware name: Bochs ovs-cloudinit-vm, BIOS Bochs 01/01/2011
> [ 4884.158003] task: ffff88038d045b00 ti: ffff880390e34000 task.ti:
> ffff880390e34000
> [ 4884.158003] RIP: 0010:[<ffffffffa006f173>]  [<ffffffffa006f173>]
> virtnet_send_command.constprop.29+0x103/0x160 [virtio_net]
> [ 4884.158003] RSP: 0018:ffff880390e379f8  EFLAGS: 00000246
> [ 4884.158003] RAX: 0000000000000000 RBX: 00000000000163e0 RCX:
> ffff88038fbd3000
> [ 4884.158003] RDX: 000000000000c0f0 RSI: ffff880390e379fc RDI:
> ffff88038e140000
> [ 4884.158003] RBP: ffff880390e37a90 R08: 00000000000163e0 R09:
> ffffffffa005059c
> [ 4884.158003] R10: 0000000000000003 R11: 0000000000000000 R12:
> 0000000000000003
> [ 4884.158003] R13: 0000000000000000 R14: 0000000000000003 R15:
> ffff880390e379e8
> [ 4884.158003] FS:  00007fc39b216740(0000) GS:ffff8803a80c0000(0000) knl
> GS:0000000000000000
> [ 4884.158003] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 4884.158003] CR2: 00007fc39b221000 CR3: 000000038cb6f000 CR4:
> 00000000000007e0
> [ 4884.158003] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 4884.158003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 4884.158003] Stack:
> [ 4884.158003]  000000000102ff46 ffff880390e37a20 ffff880390e37aa8
> ffff880390e37a40
> [ 4884.158003]  ffffffff815bb3ae ffffea000e438dc2 00000002000009fa
> 0000000000000000
> [ 4884.158003]  0000000000000000 ffffea000e438dc2 00000001000009f9
> 0000000000000000
> [ 4884.158003] Call Trace:
> [ 4884.158003]  [<ffffffff815bb3ae>] ? fib6_clean_all+0xae/0x100
> [ 4884.158003]  [<ffffffffa006f3f0>] virtnet_vlan_rx_kill_vid+0x50/0x90
> [virtio_net]
> [ 4884.158003]  [<ffffffff815e67e3>] vlan_vid_del+0x113/0x170
> [ 4884.158003]  [<ffffffffa02b681b>] vlan_device_event+0x65b/0x690 [8021q]
> [ 4884.158003]  [<ffffffff8160fdec>] notifier_call_chain+0x4c/0x70
> [ 4884.158003]  [<ffffffff8109d456>] raw_notifier_call_chain+0x16/0x20
> [ 4884.158003]  [<ffffffff814f78fd>] call_netdevice_notifiers+0x2d/0x60
> [ 4884.158003]  [<ffffffff814f7bac>] dev_close_many+0xcc/0x120
> [ 4884.158003]  [<ffffffff814f8ee8>] rollback_registered_many+0xa8/0x220
> [ 4884.158003]  [<ffffffff814f90a0>] rollback_registered+0x40/0x70
> [ 4884.158003]  [<ffffffff814fa388>] unregister_netdevice_queue+0x48/0x80
> [ 4884.158003]  [<ffffffff814fa3dc>] unregister_netdev+0x1c/0x30
> [ 4884.158003]  [<ffffffffa006e0e3>] virtnet_remove+0x43/0x80 [virtio_net]
> [ 4884.158003]  [<ffffffffa002f119>] virtio_dev_remove+0x49/0xd0 [virtio]
> [ 4884.158003]  [<ffffffff813d2cbf>] __device_release_driver+0x7f/0xf0
> [ 4884.158003]  [<ffffffff813d2d53>] device_release_driver+0x23/0x30
> [ 4884.158003]  [<ffffffff813d24c8>] bus_remove_device+0x108/0x180
> [ 4884.158003]  [<ffffffff813ce9e5>] device_del+0x135/0x1f0
> [ 4884.158003]  [<ffffffff813ceabe>] device_unregister+0x1e/0x60
> [ 4884.158003]  [<ffffffffa002f426>] unregister_virtio_device+0x16/0x30
> [virtio]
> [ 4884.158003]  [<ffffffffa01c852b>] virtio_pci_remove+0x2b/0x70
> [virtio_pci]
> [ 4884.158003]  [<ffffffff813097bb>] pci_device_remove+0x3b/0xb0
> [ 4884.158003]  [<ffffffff813d2cbf>] __device_release_driver+0x7f/0xf0
> [ 4884.158003]  [<ffffffff813d2d53>] device_release_driver+0x23/0x30
> [ 4884.158003]  [<ffffffff813d16bd>] driver_unbind+0xbd/0xe0
> [ 4884.158003]  [<ffffffff813d0bf4>] drv_attr_store+0x24/0x40
> [ 4884.158003]  [<ffffffff8123cfc6>] sysfs_write_file+0xc6/0x140
> [ 4884.158003]  [<ffffffff811c6a2d>] vfs_write+0xbd/0x1e0
> [ 4884.158003]  [<ffffffff8160fce3>] ? trace_do_page_fault+0x43/0x100
> [ 4884.158003]  [<ffffffff811c7478>] SyS_write+0x58/0xb0
> [ 4884.158003]  [<ffffffff81614409>] system_call_fastpath+0x16/0x1b
> [ 4884.158003] Code: c5 70 ff ff ff e8 2e 11 fe ff 48 8b 7b 08 e8 c5 02 fe
> ff eb 12 0f 1f 00 48 8b 7b 08 e8 77 ff fd ff 84 c0 75 17 f3 90 48 8b 7b 08
> <48> 8d b5 6c ff ff ff e8 01 01 fe ff 48 85 c0 74 dc 80 bd 69 ff
> {noformat}
>