[dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
Hu, Xuekun
xuekun.hu at intel.com
Tue Jan 15 08:07:47 CET 2019
Hi, Ravi
Did you resolve this issue that VF used in guest with vIOMMU enabled? I googled, but still can't get the answer that it is driver or qemu vt-d emulation issue.
Currently I met the same issue again that host reported DMAR error:
[59939.130110] DMAR: DRHD: handling fault status reg 2
[59939.130116] DMAR: [DMA Read] Request device [83:10.0] fault addr 15f03d000 [fault reason 06] PTE Read access is not set
[59940.180859] ixgbe 0000:83:00.0: Issuing VFLR with pending transactions
[59940.180863] ixgbe 0000:83:00.0: Issuing VFLR for VF 0000:83:10.0
[59989.344683] ixgbe 0000:83:00.0 ens817f0: VF Reset msg received from vf 0
I'm using DPDK 18.11 in guest, and ixgbe.ko in host.
Thx, Xuekun
-----Original Message-----
From: dev <dev-bounces at dpdk.org> On Behalf Of Burakov, Anatoly
Sent: Friday, February 16, 2018 5:42 PM
To: Ravi Kerur <rkerur at gmail.com>
Cc: dev at dpdk.org; Ananyev, Konstantin <konstantin.ananyev at intel.com>; Lu, Wenzhuo <wenzhuo.lu at intel.com>
Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
On 15-Feb-18 8:53 PM, Ravi Kerur wrote:
>
>
> On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur at gmail.com
> <mailto:rkerur at gmail.com>> wrote:
>
>
>
> On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly
> <anatoly.burakov at intel.com <mailto:anatoly.burakov at intel.com>> wrote:
>
> On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
>
>
> Earlier I was focusing only on DMAR errors and I might have
> said 'it worked' when I didn't notice them on host when dpdk
> was started on guest. When trying to send packets out of
> that interface from guest I did see DMAR errors. I am
> attaching information you requested. I have enabled
> log-level=8 and files contain dpdk EAL/PMD logs as well.
>
>
> Great, now we're on the same page.
>
>
> Snippets below
>
> on host, DMAR fault address from dmesg
>
> [351576.998109] DMAR: DRHD: handling fault status reg 702
> [351576.998113] DMAR: [DMA Read] Request device [04:10.0]
> fault addr 257617000 [fault reason 06] PTE Read access is
> not set
>
> on guest (dump phys_mem_layout)
>
> Segment 235: phys:0x257600000, len:2097152,
> virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152,
> nchannel:0, nrank:0
> ...
> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600
> dma_addr=0x257617600
> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80
> dma_addr=0x25406fe80
> ...
>
>
> To me this looks like host (i.e. either QEMU or the PF driver)
> is trying to do DMA using guest-physical (and not
> host-physical). I'm not too well-versed in how QEMU works, but
> i'm pretty sure that's not supposed to happen.
>
> Is PF also bound to DPDK, or are you using native Linux ixgbe
> driver?
>
>
> Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF
> interfaces disappear after it is bound to DPDK. If there is a way to
> use PF and VF with DPDK let me know I can try it out. I am not sure
> how to move forward on this, Is CPU/IXGBE PF driver playing a role?
> Following are the versions I have
>
> lscpu
> Architecture: x86_64
> CPU op-mode(s): 32-bit, 64-bit
> Byte Order: Little Endian
> CPU(s): 56
> On-line CPU(s) list: 0-27
> Off-line CPU(s) list: 28-55
> Thread(s) per core: 1
> Core(s) per socket: 14
> Socket(s): 2
> NUMA node(s): 2
> Vendor ID: GenuineIntel
> CPU family: 6
> Model: 63
> Model name: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
> Stepping: 2
> CPU MHz: 2500.610
> CPU max MHz: 3000.0000
> CPU min MHz: 1200.0000
> BogoMIPS: 4000.74
> Virtualization: VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 256K
> L3 cache: 35840K
> NUMA node0 CPU(s): 0-13
> NUMA node1 CPU(s): 14-27
>
> # ethtool -i enp4s0f0
> driver: ixgbe
> version: 5.3.3
> firmware-version: 0x800007b8, 1.1018.0
> bus-info: 0000:04:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> Thanks,
> Ravi
>
>
>
> Debugging this I could co-relate doing interface link-up associated
> with the dpdk inside the guest causes DMAR errors on host and an
> additional vflr message.
>
> [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [
> 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr
> 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074]
> ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [
> 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0
>
> Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from
> 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it
> possible that dpdk pmd vf driver is missing some fixes/porting from
> ixgbevf driver since this issue is not seen when ixgbevf kernel driver
> is used?
>
Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue.
--
Thanks,
Anatoly
More information about the dev
mailing list