unable to bind to vfio-pci

[Stephen Hemminger]

On Tue, 21 Feb 2023 16:27:31 +0000
Bruce Richardson <bruce.richardson at intel.com> wrote:

> On Tue, Feb 21, 2023 at 09:46:30PM +0530, Nagendra Prabhu A T wrote:
> >    These are the messages i am seeing dmesg
> >    [16588.298660] vfio-pci: probe of 0000:8a:00.2 failed with error -22
> >    [16591.574630] i40e 0000:8a:00.3: i40e_ptp_stop: removed PHC on
> >    enp138s0f3
> >    [16593.138613] vfio-pci: probe of 0000:8a:00.3 failed with error -22
> >    [16593.138674] vfio-pci: probe of 0000:8a:00.3 failed with error -22
> >    [16627.551866] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [16627.551922] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [16716.344530] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [16716.344585] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [17859.876566] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [17859.876622] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    root at nat-kvm:/home/nat# cat /boot/config-$(uname -r) | grep NOIOMMU
> >    CONFIG_VFIO_NOIOMMU=y
> >    root at nat-kvm:/home/nat#
> >    Interface type :
> >    Other Network devices
> >    =====================
> >    0000:51:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:51:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:51:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:51:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >   
> 
> Please try the following:
> 
> 	echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
> 	dpdk-devbind.py -b vfio-pci 8a:00.*
> 
> Does anything change then?
> 
> /Bruce
> 
> PS: For mailing list discussions, please don't top-post. Replies are easier
> to follow if posted immediately below the relevant content.
> 
> >    On Tue, Feb 21, 2023 at 9:37 PM Bruce Richardson
> >    <[1]bruce.richardson at intel.com> wrote:
> > 
> >      On Tue, Feb 21, 2023 at 09:30:15PM +0530, Nagendra Prabhu A T wrote:  
> >      >    HI Bruce,
> >      >    Thanks for your reply.
> >      >    I have removed the binding of all other ports. So , i can  
> >      confirm that  
> >      >    #2 is taken care of.
> >      >    I tried using a different driver "uio_pci_generic". I still see  
> >      the  
> >      >    same issue for that driver as well.
> >      >    The problem seems to be because the write operation is not  
> >      happening in  
> >      >    "bind" file.
> >      >    Infact tried doing the following step manually. That also  
> >      failed  
> >      >    echo sh -c 0000:51:00.0 >  
> >      /sys/bus/pci/drivers/uio_pci_generic/bind  
> >      >    bash: echo: write error: No such device  
> >      The "sh -c" doesn't look right in that command. It should just echo
> >      the
> >      device id to the bind file. However, it's generally recommended just
> >      to use
> >      the devbind script, as more writes to sysfs may be necessary first,
> >      e.g. to
> >      register the device ids first.  
> >      >    I have required permission for the file.
> >      >    I tried doing with "sudo"
> >      >    I dont have selinux running
> >      >    So not sure whats blocking me. Infact , even
> >      >    "cat /sys/bus/pci/drivers/uio_pci_generic/bind" is failing
> >      >    Can you please help.  
> >      To track down issues other than the obvious, I think we need more
> >      information. For example, what NIC card is this? Are there any
> >      useful error
> >      messages in dmesg?
> >      /Bruce
> > 
> > References
> > 
> >    1. mailto:bruce.richardson at intel.com  

Another possibility is that the hardware shares IOMMU channels.
One machine I was using had multiple PCI devices sharing same IOMMU channel and kernel
would not let it work unless all of them were unbound from kernel.