l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)

F32 feng32 at 163.com
Tue Sep 20 13:04:57 CEST 2022
Previous message (by thread): l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
Next message (by thread): l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi Stephen,


I have checked related iommu_groups:


$ readlink 0000\:01\:00.0/iommu_group

../../../../kernel/iommu_groups/16

$ readlink 0000\:02\:00.0/iommu_group

../../../../kernel/iommu_groups/17

$ readlink 0000\:03\:00.0/iommu_group

../../../../kernel/iommu_groups/18

$ readlink 0000\:04\:00.0/iommu_group

../../../../kernel/iommu_groups/19

$ readlink 0000\:05\:00.0/iommu_group

../../../../kernel/iommu_groups/20



It seems that the adapters belongs to different iommu_groups.


At 2022-09-16 23:10:59, "Stephen Hemminger" <stephen at networkplumber.org> wrote:
>On Wed, 14 Sep 2022 11:18:11 +0800 (CST)
>F32 <feng32 at 163.com> wrote:
>
>> Hi,
>> 
>> 
>> Recently I'm testing dpdk on a mini PC with N5095 CPU and 5 I225-V ethernet adapters under Ubuntu Server 22.04.1.
>> 
>> 
>> $ uname -r
>> 5.15.0-47-generic
>> 
>> 
>> $ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
>> GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt"
>> $ update-grub
>> $ reboot
>> 
>> 
>> $ dmesg | grep DMAR
>> [    0.016399] ACPI: DMAR 0x00000000787D6000 000088 (v02 INTEL  EDK2     00000002      01000013)
>> [    0.016443] ACPI: Reserving DMAR table memory at [mem 0x787d6000-0x787d6087]
>> [    0.029273] DMAR: IOMMU enabled
>> [    0.093267] DMAR: Host address width 39
>> [    0.093269] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
>> [    0.093276] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 49e2ff0505e
>> [    0.093281] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
>> [    0.093287] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
>> [    0.093291] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fbfffff
>> [    0.093296] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
>> [    0.093299] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
>> [    0.093301] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
>> [    0.095158] DMAR-IR: Enabled IRQ remapping in x2apic mode
>> [    0.288306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
>> [    0.372113] DMAR: No ATSR found
>> [    0.372115] DMAR: No SATC found
>> [    0.372118] DMAR: IOMMU feature fl1gp_support inconsistent
>> [    0.372120] DMAR: IOMMU feature pgsel_inv inconsistent
>> [    0.372122] DMAR: IOMMU feature nwfs inconsistent
>> [    0.372124] DMAR: IOMMU feature pds inconsistent
>> [    0.372125] DMAR: IOMMU feature eafs inconsistent
>> [    0.372127] DMAR: IOMMU feature prs inconsistent
>> [    0.372129] DMAR: IOMMU feature nest inconsistent
>> [    0.372130] DMAR: IOMMU feature mts inconsistent
>> [    0.372132] DMAR: IOMMU feature sc_support inconsistent
>> [    0.372134] DMAR: IOMMU feature dev_iotlb_support inconsistent
>> [    0.372136] DMAR: dmar0: Using Queued invalidation
>> [    0.372141] DMAR: dmar1: Using Queued invalidation
>> [    0.373075] DMAR: Intel(R) Virtualization Technology for Directed I/O
>> 
>> 
>> $ lspci
>> lspci
>> 00:00.0 Host bridge: Intel Corporation Device 4e24
>> 00:02.0 VGA compatible controller: Intel Corporation JasperLake [UHD Graphics] (rev 01)
>> 00:04.0 Signal processing controller: Intel Corporation Dynamic Tuning service
>> 00:08.0 System peripheral: Intel Corporation Device 4e11
>> 00:14.0 USB controller: Intel Corporation Device 4ded (rev 01)
>> 00:14.2 RAM memory: Intel Corporation Device 4def (rev 01)
>> 00:15.0 Serial bus controller: Intel Corporation Serial IO I2C Host Controller (rev 01)
>> 00:15.2 Serial bus controller: Intel Corporation Device 4dea (rev 01)
>> 00:16.0 Communication controller: Intel Corporation Management Engine Interface (rev 01)
>> 00:17.0 SATA controller: Intel Corporation Device 4dd3 (rev 01)
>> 00:19.0 Serial bus controller: Intel Corporation Device 4dc5 (rev 01)
>> 00:19.1 Serial bus controller: Intel Corporation Device 4dc6 (rev 01)
>> 00:1c.0 PCI bridge: Intel Corporation Device 4db8 (rev 01)
>> 00:1c.4 PCI bridge: Intel Corporation Device 4dbc (rev 01)
>> 00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01)
>> 00:1c.6 PCI bridge: Intel Corporation Device 4dbe (rev 01)
>> 00:1c.7 PCI bridge: Intel Corporation Device 4dbf (rev 01)
>> 00:1e.0 Communication controller: Intel Corporation Device 4da8 (rev 01)
>> 00:1e.3 Serial bus controller: Intel Corporation Device 4dab (rev 01)
>> 00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)
>> 00:1f.3 Audio device: Intel Corporation Device 4dc8 (rev 01)
>> 00:1f.4 SMBus: Intel Corporation JaserLake SMBus (rev 01)
>> 00:1f.5 Serial bus controller: Intel Corporation JaserLake SPI (flash) Controller (rev 01)
>> 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>> 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>> 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>> 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>> 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>> 
>> 
>> (after huge page setup)
>> $ python3 dpdk-hugepages.py -s
>> Node Pages Size Total
>> 0    1607  2Mb    3Gb
>> 
>> 
>> Hugepages mounted on /dev/hugepages /mnt/huge
>> 
>> 
>> $ ifconfig enp2s0 down
>> $ ifconfig enp3s0 down
>> $ python3 dpdk-devbind.py --bind=vfio-pci 0000:02:00.0
>> $ python3 dpdk-devbind.py --bind=vfio-pci 0000:03:00.0
>> $ python3 dpdk-devbind.py -s
>> 
>> 
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:02:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
>> 0000:03:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
>> 
>> 
>> Network devices using kernel driver
>> ===================================
>> 0000:01:00.0 'Ethernet Controller I225-V 15f3' if=enp1s0 drv=igc unused=vfio-pci *Active*
>> 0000:04:00.0 'Ethernet Controller I225-V 15f3' if=enp4s0 drv=igc unused=vfio-pci 
>> 0000:05:00.0 'Ethernet Controller I225-V 15f3' if=enp5s0 drv=igc unused=vfio-pci 
>> 
>> 
>> No 'Baseband' devices detected
>> ==============================
>> 
>> 
>> No 'Crypto' devices detected
>> ============================
>> 
>> 
>> No 'DMA' devices detected
>> =========================
>> 
>> 
>> No 'Eventdev' devices detected
>> ==============================
>> 
>> 
>> No 'Mempool' devices detected
>> =============================
>> 
>> 
>> No 'Compress' devices detected
>> ==============================
>> 
>> 
>> No 'Misc (rawdev)' devices detected
>> ===================================
>> 
>> 
>> No 'Regex' devices detected
>> ===========================
>> 
>> 
>> After the above configuration, dpck setup should be done. Then I tried to run l2fwd. It generates the following errror messages:
>> 
>> 
>> $ ./l2fwd
>> EAL: Detected CPU lcores: 4
>> EAL: Detected NUMA nodes: 1
>> EAL: Detected shared linkage of DPDK
>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> EAL: Selected IOVA mode 'VA'
>> EAL: VFIO support initialized
>> EAL: Using IOMMU type 1 (Type 1)
>> EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:02:00.0 (socket 0)
>> EAL: Releasing PCI mapped resource for 0000:02:00.0
>> EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100800000
>> EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100900000
>> EAL: Requested device 0000:02:00.0 cannot be used
>> EAL: Using IOMMU type 1 (Type 1)
>> EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:03:00.0 (socket 0)
>> EAL: Releasing PCI mapped resource for 0000:03:00.0
>> EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100904000
>> EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100a04000
>> EAL: Requested device 0000:03:00.0 cannot be used
>> TELEMETRY: No legacy callbacks, legacy socket not created
>> MAC updating enabled
>> EAL: Error - exiting with code: 1
>>   Cause: No Ethernet ports - bye
>> 
>> 
>> It seems that the I225 adapters are not recognized by DPDK. I can not see any detailed reasons why the adapters cannot be used.
>> Any suggestions to fix this issue?
>> 
>> 
>> Regards,
>> F32
>
>Check the IOMMU groups on that system.
>Often multiple PCI devices (especially multi-port NICs) share a single IOMMU
>group and VFIO does not allow using these devices unless all subdevices
>in the group are unbound from the kernel driver.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mails.dpdk.org/archives/users/attachments/20220920/715f6bae/attachment-0001.htm>
Previous message (by thread): l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
Next message (by thread): l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the users mailing list