[dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
Burakov, Anatoly
anatoly.burakov at intel.com
Mon Feb 5 11:01:41 CET 2018
On 02-Feb-18 8:51 PM, Ravi Kerur wrote:
>
>
> On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rkerur at gmail.com
> <mailto:rkerur at gmail.com>> wrote:
>
>
>
> On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly
> <anatoly.burakov at intel.com <mailto:anatoly.burakov at intel.com>> wrote:
>
> On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
>
>
>
> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly
> <anatoly.burakov at intel.com
> <mailto:anatoly.burakov at intel.com>
> <mailto:anatoly.burakov at intel.com
> <mailto:anatoly.burakov at intel.com>>> wrote:
>
> On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
>
>
> Hi Anatoly,
>
> Thanks. I am following wiki link below which uses
> vIOMMU with
> DPDK as a use-case and instantiate VM as specified
> with Q35
> chipset in Qemu.
>
> https://wiki.qemu.org/Features/VT-d
> <https://wiki.qemu.org/Features/VT-d>
> <https://wiki.qemu.org/Features/VT-d
> <https://wiki.qemu.org/Features/VT-d>>
>
> Qemu-version is 2.11
> Host kernel 4.9
> Guest kernel 4.4
>
> I can only guess that guest kernel needs an upgrade
> in my setup
> to work correctly, if versions on my setup rings a
> bell on not
> having support kindly let me know.
>
> When 'modprobe vfio enable_unsafe_noiommu_node=Y'
> is executed on
> guest I get following error
> ...
> vfio: unknown parameter
> 'enable_unsafe_noiommu_node' ignored
> ...
>
> in guest.
>
> Thanks.
>
>
> AFAIK kernel 4.4 should have noiommu mode - it was
> introduced in
> 3.1x days. However, in order for that to work, kernel
> also has to be
> built with this mode enabled. My guess is, whoever is
> the supplier
> of your kernel, did not do that. You should
> double-check the kernel
> configuration of your distribution.
>
> However, if you have vIOMMU in QEMU, you shouldn't need
> noiommu mode
> - "regular" vfio should work fine. noiommu mode should
> only be
> needed if you know you don't have IOMMU enabled in your
> kernel, and
> even if you can't enable it, you can still use igb_uio.
>
> Hi Anatoly,
>
> Do you suggest I take this discussion to kvm/qemu mailing
> list as I am not sure which component has the issue? I check
> dmesg for BIOS physical memory map and address reported as
> fault by DMAR is reported by BIOS as usable on both host and vm.
>
> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0]
> fault addr *33a128000 *[fault reason 06] PTE Read access is
> not set
>
> dmesg | grep BIOS
> [ 0.000000] e820: BIOS-provided physical RAM map:
> [ 0.000000] BIOS-e820: [mem
> 0x0000000000000000-0x000000000009afff] usable
> [ 0.000000] BIOS-e820: [mem
> 0x000000000009b000-0x000000000009ffff] reserved
> [ 0.000000] BIOS-e820: [mem
> 0x00000000000e0000-0x00000000000fffff] reserved
> [ 0.000000] BIOS-e820: [mem
> 0x0000000000100000-0x000000007938afff] usable
> [ 0.000000] BIOS-e820: [mem
> 0x000000007938b000-0x000000007994bfff] reserved
> [ 0.000000] BIOS-e820: [mem
> 0x000000007994c000-0x000000007999cfff] ACPI data
> [ 0.000000] BIOS-e820: [mem
> 0x000000007999d000-0x0000000079f7dfff] ACPI NVS
> [ 0.000000] BIOS-e820: [mem
> 0x0000000079f7e000-0x000000007bd37fff] reserved
> [ 0.000000] BIOS-e820: [mem
> 0x000000007bd38000-0x000000007bd38fff] usable
> [ 0.000000] BIOS-e820: [mem
> 0x000000007bd39000-0x000000007bdbefff] reserved
> [ 0.000000] BIOS-e820: [mem
> 0x000000007bdbf000-0x000000007bffffff] usable
> [ 0.000000] BIOS-e820: [mem
> 0x000000007c000000-0x000000008fffffff] reserved
> [ 0.000000] BIOS-e820: [mem
> 0x00000000fed1c000-0x00000000fed44fff] reserved
> [ 0.000000] BIOS-e820: [mem
> 0x00000000ff000000-0x00000000ffffffff] reserved
> [* 0.000000] BIOS-e820: [mem
> 0x0000000100000000-0x000000407fffffff] usable*
> *
> *
> Kindly let me know your inputs.
>
> Thanks.
>
>
> -- Thanks,
> Anatoly
>
>
>
> The "PTE Read not set" error usually indicates that you are
> trying to use a non-IOMMU method when you have IOMMU enabled
> (i.e. trying to use igb_uio when IOMMU is on). That, to me,
> indicates that you do have IOMMU emulation enabled.
>
> I would go about it this way.
>
> First, i'd ensure that your VM has IOMMU emulation enabled and
> working. You have mentioned that your QEMU version should have
> IOMMU emulation, so let's assume that's the case.
>
> I am not sure of the exact command-line needed to activate the
> vIOMMU emulation, but assuming your VM emulates an Intel
> processor, your kernel command-line should have "iommu=on
> intel_iommu=on" in it. Check /etc/default/grub for
> GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are
> not in there, add the above changes, do "update-grub" and reboot
> your VM.
>
> If it already did have the necessary kernel configuration, do
> "dmesg | grep IOMMU" and look for "IOMMU Enabled". That should
> tell you that IOMMU is enabled and working in the kernel.
>
> After that, you can modprobe vfio and vfio-pci, bind NICs to it,
> and it should be working. Please bear in mind that all of that
> is how i would've gone about it if i had similar problems on
> baremetal, but i'm hoping all of it is applicable to VM's. So,
> either disable IOMMU and use igb_uio, or enable IOMMU and use
> VFIO. Both will work.
>
>
> Thanks for your help so far on this. I have everything enabled as
> you have mentioned in your email. Below is the error from EAL when
> DPDK is run on a VM. I am referring EAL code to check why it is failing.
>
> ************VM DPDK errors***********
> EAL: Detected 14 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> unreliable clock cycles !
> EAL: PCI device 0000:00:07.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:10ed net_ixgbe_vf
> EAL: using IOMMU type 1 (Type 1)
> *EAL: RTE_IOVA_VA 140051367329792
> EAL: cannot set up DMA remapping, error 14 (Bad address)
> EAL: 0000:00:07.0 DMA remapping failed, error 14 (Bad address)*
> EAL: Requested device 0000:00:07.0 cannot be used
> EAL: PCI device 0000:03:00.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 1af4:1041 net_virtio
> EAL: PCI device 0000:04:00.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 1af4:1041 net_virtio
> EAL: PCI device 0000:05:00.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 1af4:1041 net_virtio
> EAL: No probed ethernet devices
> Interactive-mode selected
> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
> size=2176, socket=0
>
>
> ********************VM************
> root at sriov-ravi-01:~# lsmod | grep vfio
> *vfio_pci 40960 1
> vfio_virqfd 16384 1 vfio_pci
> vfio_iommu_type1 20480 1
> vfio 28672 5 vfio_iommu_type1,vfio_pci
> irqbypass 16384 3 kvm,vfio_pci*
> root at sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU
> [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS
> BXPCDMAR 00000001 BXPC 00000001)
> *[ 0.000000] DMAR: IOMMU enabled*
> [ 9.423839] DMAR: Host address width 39
> [ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
> [ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
> 12008c22260286 ecap f00f5e
> [ 9.427314] DMAR: ATSR flags: 0x1
> [ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0
> [ 9.429255] DMAR-IR: Queued invalidation will be enabled to
> support x2apic and Intr-remapping.
> [ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [ 11.167334] DMAR: No RMRR found
> [ 11.168855] DMAR: dmar0: Using Queued invalidation
> [ 12.119060] DMAR: Setting RMRR:
> [ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC
> [ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0
> [0x0 - 0xffffff]
> [ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O
> root at sriov-ravi-01:~# dpdk-devbind -s
>
> Network devices using DPDK-compatible driver
> ============================================
> *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed'
> drv=vfio-pci unused=*
>
> Network devices using kernel driver
> ===================================
> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci
> *Active*
> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
>
> Other Network devices
> =====================
> <none>
>
> Crypto devices using DPDK-compatible driver
> ===========================================
> <none>
>
> Crypto devices using kernel driver
> ==================================
> <none>
>
> Other Crypto devices
> ====================
> <none>
>
> Eventdev devices using DPDK-compatible driver
> =============================================
> <none>
>
> Eventdev devices using kernel driver
> ====================================
> <none>
>
> Other Eventdev devices
> ======================
> <none>
>
> Mempool devices using DPDK-compatible driver
> ============================================
> <none>
>
> Mempool devices using kernel driver
> ===================================
> <none>
>
> Other Mempool devices
> =====================
> <none>
> root at sriov-ravi-01:~#
>
> root at sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg |
> grep -i vfio
> [ 77.676711] VFIO - User Level meta-driver version: 0.3
> [ 149.013806] VFIO - User Level meta-driver version: 0.3
> [ 275.876440] VFIO - User Level meta-driver version: 0.3
>
> Thanks,
> Ravi
>
>
>
> Additonal information. On VM following message is seen
>
> [ 298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient
> for the mapped address (7f7300000000)
> [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient
> for the mapped address (7fa580000000)
>
> ON VM, DMAR reports address width = 39
> root at sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep
> -e IOMMU -e DMAR
> [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR
> 00000001 BXPC 00000001)
> [ 0.000000] DMAR: IOMMU enabled
> [ 9.209145] DMAR: Host address width 39
>
> On Host, DMAR reports address width = 46
> # dmesg | grep -e DMAR -e IOMMU
> [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
> 00000001 INTL 20091013)
> [ 0.000000] DMAR: IOMMU enabled
> [ 0.519368] DMAR: Host address width 46
> [ 0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>
> Searching for IOMMU errors found following link from RedHat which points
> to DPDK 17.11 issue.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1530957
>
> Thanks.
>
> --
> Thanks,
> Anatoly
>
>
>
Ah, yes, that. We have recently applied a patch dealing with exactly this:
http://dpdk.org/dev/patchwork/patch/33650/
Unfortunately, as far as i know, other than refusing to run (like the
patch does), there's no way to fix this issue, so you'll have to disable
IOMMU in your VM, and use either igb_uio or vfio in noiommu mode.
--
Thanks,
Anatoly
More information about the dev
mailing list