[dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Burakov, Anatoly]

On 29-Jan-18 10:35 PM, Ravi Kerur wrote:
> Hi Burakov,
> 
> When using vfio-pci on host both VF and PF interfaces works fine with 
> dpdk i.e. I don't see DMAR fault messages anymore. However, when I 
> attach a VF interface to a VM and start DPDK with vfio-pci inside VM I 
> still see DMAR fault messages on host. Both host and VM are booted with 
> 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work 
> (I think it's expected because of DMAR faults), however, when VF 
> interface uses ixgbevf driver ping works.
> 
> Following are some details
> 
> /*****************On VM***************/
> dpdk-devbind -s
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci 
> unused=ixgbevf
> 
> Network devices using kernel driver
> ===================================
> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
> 
> Other network devices
> =====================
> <none>
> 
> Crypto devices using DPDK-compatible driver
> ===========================================
> <none>
> 
> Crypto devices using kernel driver
> ==================================
> <none>
> 
> Other crypto devices
> ====================
> <none>
> 
> 
> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller 
> Virtual Function (rev 01)
>          Subsystem: Intel Corporation 82599 Ethernet Controller Virtual 
> Function
>          Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- 
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>          Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K]
>          Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K]
>          Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>                  Vector table: BAR=3 offset=00000000
>                  PBA: BAR=3 offset=00002000
>          Capabilities: [a0] Express (v1) Root Complex Integrated 
> Endpoint, MSI 00
>                  DevCap: MaxPayload 128 bytes, PhantFunc 0
>                          ExtTag- RBE-
>                  DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
> Unsupported-
>                          RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                          MaxPayload 128 bytes, MaxReadReq 128 bytes
>                  DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- 
> AuxPwr- TransPend-
>          Capabilities: [100 v1] Advanced Error Reporting
>                  UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                  UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                  UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                  CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
> NonFatalErr-
>                  CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
> NonFatalErr-
>                  AERCap: First Error Pointer: 00, GenCap- CGenEn- 
> ChkCap- ChkEn-
>          Kernel driver in use: vfio-pci
>          Kernel modules: ixgbevf
> 
> /***************on Host*************/
> dmesg | grep DMAR
> ...
> [  978.268143] DMAR: DRHD: handling fault status reg 2
> [  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> [ 1286.677726] DMAR: DRHD: handling fault status reg 102
> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> fb663000 [fault reason 06] PTE Read access is not set
> [ 1676.436145] DMAR: DRHD: handling fault status reg 202
> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> [ 1734.433649] DMAR: DRHD: handling fault status reg 302
> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> [ 2324.428938] DMAR: DRHD: handling fault status reg 402
> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 7770c000 [fault reason 06] PTE Read access is not set
> [ 2388.553640] DMAR: DRHD: handling fault status reg 502
> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> 
> 

Going back to this, i would like to suggest run a few tests to ensure 
that we have all information that we can gather.

First of all, i'm assuming that you're using native ixgbe Linux driver 
on the host, and that you're only passing through the VF device to the 
VM using VFIO. Is my understanding correct here?

Now, let's forget about the iommu=pt and igb_uio for a moment. Boot both 
your host and your VM with iommu=on and intel_iommu=on (or whatever 
command-line enables full IOMMU support on both host and guest) and do 
the same tests you've done before. Do you still see your issues?

It would also be very useful to also try native Linux kernel driver on 
the guest *with traffic forwarding* and see how it works in your VM. 
Therefore i would suggest you to compile DPDK with PCAP support, bind 
your (VM) interface to native Linux driver, and use the interface via 
our pcap driver (creating a vdev should do the trick - please refer to 
PCAP PMD documentation [1]). Simple forwarding test should be enough - 
just make sure to pass traffic to and from DPDK in both cases, and that 
it doesn't give you any DMAR errors.

We can go from there.

[1] http://dpdk.org/doc/guides/nics/pcap_ring.html

-- 
Thanks,
Anatoly