[dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Stojaczyk, Dariusz <dariusz.stojaczyk at intel.com>
> Sent: Tuesday, July 23, 2019 10:24 AM
> To: Jerin Jacob Kollanukkaran <jerinj at marvell.com>; Thomas Monjalon
> <thomas at monjalon.net>; David Marchand <david.marchand at redhat.com>;
> Burakov, Anatoly <anatoly.burakov at intel.com>
> Cc: dev at dpdk.org
> Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection
> 
> > -----Original Message-----
> > From: Jerin Jacob Kollanukkaran [mailto:jerinj at marvell.com]
> > Sent: Tuesday, July 23, 2019 6:19 AM
> >
> > > -----Original Message-----
> > > From: Stojaczyk, Dariusz <dariusz.stojaczyk at intel.com>
> > > Sent: Tuesday, July 23, 2019 9:06 AM
> > > To: Thomas Monjalon <thomas at monjalon.net>; David Marchand
> > > <david.marchand at redhat.com>; Burakov, Anatoly
> > > <anatoly.burakov at intel.com>; Jerin Jacob Kollanukkaran
> > > <jerinj at marvell.com>
> > > Cc: dev at dpdk.org
> > > Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode
> > > selection
> > >
> > > This introduces a regression where uio-bound devies are attached to
> > > a
> > DPDK
> > > app at runtime.
> >
> > Just to understand the requirements;
> > # Is this requirement for SPDK?
> > # Is brand new PCI device scanned and attached to DPDK at runtime?
> > # Any specific reason for using uio vs vfio?
> 
> Jerin,

Stojaczyk,

There reason to choose VA incase if bus detects DC is following:

- All drivers are expected to work in RTE_IOVA_VA mode, irrespective of
      physical address availability.
- By default, the mempool, first asks for IOVA-contiguous memory using
      ``RTE_MEMZONE_IOVA_CONTIG``. This is slow in RTE_IOVA_PA mode and it may
      affect the application boot time.
- It is easy to enable large amount of IOVA-contiguous memory use-cases
      with IOVA in VA mode.

> 
> It came up in SPDK tests, but it's certainly nothing SPDK-specific, I can't give
> you the steps but it should be reproducible even with testpmd.
> 
> The PCI device could have been simply hotplugged to the system after DPDK
> app start. DPDK didn't know about it at initialization, so it picked
> RTE_IOVA_VA and then would fail to attach any UIO-bound device ever
> after:
> 
> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing

We have RTE_PCI_DRV_NEED_IOVA_AS_VA devices in DPDK, Which can work
Only on VA. If we default 'PA' incase of DC, then what do with hotplugging on those devices?


> EAL: Driver cannot attach the device (0000:00:09.0)
> EAL: Failed to attach device on primary process
> 
> UIO is commonly used on systems without IOMMU- including VMs.

The latest machines has IOMMU. Which machines you are testing against,
Can we detect the machines without IOMMU and switch to PA?

> 
> >
> > If it is for SPDK,
> > # How about introducing rte_eal_init_with_mode(enum rte_iova_mode)?
> > # How about adding dummy bus which returns RTE_IOVA_PA in the
> > bus_get_iommus_class() in SPDK code base?
> 
> There's already an --iova=mode option in DPDK that forces the iova mode.
> I'm not concerned about configurability, but the regression in the default
> behavior.
> 
> I can add workarounds to SPDK, sure, but that wouldn't be a very healthy
> approach.

Nothing like workaround, I am looking for the options for expressing
The requirements for PA?


> 
> D.
> 
> >
> > >
> > > When there are no devices attached at initialization, the only safe
> > > default should be RTE_IOVA_PA. With RTE_IOVA_VA we just won't be
> > > able to do any DMA to uio-bound PCI devices.
> > >
> > > Can we revert this patch?
> > >
> > > D.
> > >
> > > > -----Original Message-----
> > > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Thomas
> > Monjalon
> > > > Sent: Monday, July 22, 2019 5:53 PM
> > > > To: David Marchand <david.marchand at redhat.com>; Burakov, Anatoly
> > > > <anatoly.burakov at intel.com>; jerinj at marvell.com
> > > > Cc: dev at dpdk.org
> > > > Subject: Re: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode
> > > > selection
> > > >
> > > > 22/07/2019 14:56, David Marchand:
> > > > > Following the issues reported by Jerin and the discussion that
> > > > > emerged from it, here are fixes to restore and document the
> > > > > behavior of the EAL and the pci bus driver.
> > > > >
> > > > > I pondered all the arguments and tried to have the less changes
> > > > > possible.
> > > > > I can't find a need for a flag to just announce support of
> > > > > physical addresses from the pmd point of view.
> > > > > So it ended up with something really close to what Jerin had
> suggested.
> > > > >
> > > > > But the problem is that this is still unfinished wrt the documentation.
> > > > > I will be offline for 10 days and we need this to move forward,
> > > > > so sending anyway.
> > > > >
> > > > > Changelog since v3:
> > > > > - fixed typos in patch 2,
> > > > > - updated patch 3 title,
> > > > > - moved and reworded comments in the note section in patch 4,
> > > > >
> > > > > Changelog since v2 (Jerin):
> > > > > - Patch 2/4 - Remove personal appeals in log messages(Anatoly)
> > > > > - Patch 4/4 - Added documentation (Anatoly)
> > > > >
> > > > > Changelog since v1 (Jerin):
> > > > > - Changed RTE_PCI_DRV_IOVA_AS_VA flag name as
> > > > RTE_PCI_DRV_NEED_IOVA_AS_VA
> > > > >   (patch 3/4)
> > > > > - Changed IOVA mode as VA for default case(patch 4/4) with
> > > > documentation
> > > > > - Tested the patch series on octeontx2 platform
> > > >
> > > > Applied, thanks Jerin, Anatoly and David for converging on a
> > > > documented solution together.
> > > >
> > > >