[dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file

[Yang, Qiming]

Hi, Thomas
We meet a urgent issue because of Shijith's patch. 
When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe and qede(Harish said) VF all meet initial failure.
This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all works well.
Considering this issue will block our vf driver develop and test work, could we revert this patch first and find another appropriate way?

Qiming

> -----Original Message-----
> From: Patil, Harish [mailto:Harish.Patil at cavium.com]
> Sent: Thursday, September 14, 2017 3:45 AM
> To: Yigit, Ferruh <ferruh.yigit at intel.com>; Hu, Xuekun <xuekun.hu at intel.com>;
> Thotton, Shijith <Shijith.Thotton at cavium.com>; Yang, Qiming
> <qiming.yang at intel.com>; Gregory Etelson <gregory at weka.io>
> Cc: dev at dpdk.org; Tan, Jianfeng <jianfeng.tan at intel.com>
> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
> and release of device file
> 
> 
> 
> -----Original Message-----
> From: dev <dev-bounces at dpdk.org> on behalf of Ferruh Yigit
> <ferruh.yigit at intel.com>
> Date: Wednesday, September 13, 2017 at 10:06 AM
> To: "Hu, Xuekun" <xuekun.hu at intel.com>, "Thotton, Shijith"
> <Shijith.Thotton at cavium.com>, "Yang, Qiming" <qiming.yang at intel.com>,
> Gregory Etelson <gregory at weka.io>
> Cc: "dev at dpdk.org" <dev at dpdk.org>, "Tan, Jianfeng" <jianfeng.tan at intel.com>
> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
> and release of device file
> 
> >On 9/13/2017 3:25 PM, Hu, Xuekun wrote:
> >> I met the same issue too, only with i40e 2.1.26 PF kernel driver.
> >>
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Ferruh Yigit
> >> Sent: Wednesday, September 13, 2017 7:04 PM
> >> To: Shijith Thotton <shijith.thotton at caviumnetworks.com>; Yang,
> >>Qiming <qiming.yang at intel.com>; Gregory Etelson <gregory at weka.io>
> >> Cc: dev at dpdk.org; Tan, Jianfeng <jianfeng.tan at intel.com>
> >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR
> >>during open and release of device file
> >>
> >> On 9/13/2017 11:48 AM, Shijith Thotton wrote:
> >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote:
> >>>>    Hi, Shijith
> >>>>
> >>>>
> >>>>
> >>>>    VF init error will happen after apply your patch, error log as
> >>>>below. If
> >>>>    revert your commit, all things work well. And this issue is not
> >>>>only occur
> >>>>    in i40 VF but also ixgbe.
> >
> >Hi Qiming,
> >
> >I can reproduce the issue for the case:
> >
> >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k)
> >2- DPDK app run on host, so both PF and VF are in host. Linux driver
> >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and
> >DPDK VF on guest, this works fine)
> >
> >And I confirm the function igbuio_pci_open() added with below patch is
> >causing it.
> >
> >igbuio_pci_open() does:
> >    pci_reset_function(dev);
> >    pci_set_master(dev);
> >
> >Don't know yet root cause of the error, will dig more.
> >
> >>>>
> >>>>    Could you help to check it soon?
> >>>>
> >>>>
> >>>>
> >>>>    [root at localhost app]# ./testpmd -c 7 -n 4 -- -i
> >>>>
> >>>>    EAL: Detected 10 lcore(s)
> >>>>
> >>>>    EAL: No free hugepages reported in hugepages-1048576kB
> >>>>
> >>>>    EAL: Probing VFIO support...
> >>>>
> >>>>    EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> >>>>    unreliable clock cycles !
> >>>>
> >>>>    EAL: PCI device 0000:00:03.0 on NUMA socket -1
> >>>>
> >>>>    EAL:   Invalid NUMA socket, default to 0
> >>>>
> >>>>    EAL:   probe driver: 8086:154c net_i40e_vf
> >>>>
> >>>>    i40evf_init_vf(): init_adminq failed: -53
> >>>>
> >>>>    i40evf_dev_init(): Init vf failed
> >>>>
> >>>>    EAL: Requested device 0000:00:03.0 cannot be used
> >>>>
> >>>>
> >>>>
> >>>>    commit b58eedfc7dd57eef6d12e2c654a52c834f36084a
> >>>>    Author: Shijith Thotton <shijith.thotton at caviumnetworks.com>
> >>>>    Date: Fri Jul 7 16:43:51 2017 +0530
> >>>>
> >>>>    igb_uio: issue FLR during open and release of device file
> >>>>
> >>>>    Set UIO info device file operations open and release. Call pci
> >>>>reset
> >>>>    function inside open and release to clear device state at start
> >>>>and end.
> >>>>    Copied this behaviour from vfio_pci kernel module code. With
> >>>>this patch,
> >>>>    it is not mandatory to issue FLR by PMD's during init and close.
> >>>>
> >>>>    Bus master enable and disable are added in open and release
> >>>>respectively
> >>>>    to take care of device DMA.
> >>>>
> >>>>    Signed-off-by: Shijith Thotton <shijith.thotton at caviumnetworks.com>
> >>>>    Reviewed-by: Jianfeng Tan <jianfeng.tan at intel.com>
> >>>>    Acked-by: Ferruh Yigit <ferruh.yigit at intel.com>
> >>>>    Acked-by: Gregory Etelson <gregory at weka.io>
> >>>>
> >>>>
> >>>>
> >>>>    Best Regard,
> >>>>
> >>>>    Yang Qiming
> >>>>
> >>>>
> >>>
> >>> Hi Yang Qiming,
> >>>
> >>> We didn't face this issue while preparing patch. I think, Gregory
> >>> tested it for Intel NICs. Adding more people to the loop for help.
> >>>
> >>
> >> I will try to reproduce the issue.
> >>
> >>> Shijith
> 
> [Harish] We are seeing similar issue with the bringup of qede VF devices due to
> this patch.
> Could we revert this and leave it to individual PMDs to manage it?
> >>