[dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection
Burakov, Anatoly
anatoly.burakov at intel.com
Tue Jul 23 11:57:33 CEST 2019
On 23-Jul-19 6:27 AM, Jerin Jacob Kollanukkaran wrote:
>> -----Original Message-----
>> From: Stojaczyk, Dariusz <dariusz.stojaczyk at intel.com>
>> Sent: Tuesday, July 23, 2019 10:24 AM
>> To: Jerin Jacob Kollanukkaran <jerinj at marvell.com>; Thomas Monjalon
>> <thomas at monjalon.net>; David Marchand <david.marchand at redhat.com>;
>> Burakov, Anatoly <anatoly.burakov at intel.com>
>> Cc: dev at dpdk.org
>> Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection
>>
>>> -----Original Message-----
>>> From: Jerin Jacob Kollanukkaran [mailto:jerinj at marvell.com]
>>> Sent: Tuesday, July 23, 2019 6:19 AM
>>>
>>>> -----Original Message-----
>>>> From: Stojaczyk, Dariusz <dariusz.stojaczyk at intel.com>
>>>> Sent: Tuesday, July 23, 2019 9:06 AM
>>>> To: Thomas Monjalon <thomas at monjalon.net>; David Marchand
>>>> <david.marchand at redhat.com>; Burakov, Anatoly
>>>> <anatoly.burakov at intel.com>; Jerin Jacob Kollanukkaran
>>>> <jerinj at marvell.com>
>>>> Cc: dev at dpdk.org
>>>> Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode
>>>> selection
>>>>
>>>> This introduces a regression where uio-bound devies are attached to
>>>> a
>>> DPDK
>>>> app at runtime.
>>>
>>> Just to understand the requirements;
>>> # Is this requirement for SPDK?
>>> # Is brand new PCI device scanned and attached to DPDK at runtime?
>>> # Any specific reason for using uio vs vfio?
>>
>> Jerin,
>
> Stojaczyk,
>
> There reason to choose VA incase if bus detects DC is following:
>
> - All drivers are expected to work in RTE_IOVA_VA mode, irrespective of
> physical address availability.
> - By default, the mempool, first asks for IOVA-contiguous memory using
> ``RTE_MEMZONE_IOVA_CONTIG``. This is slow in RTE_IOVA_PA mode and it may
> affect the application boot time.
> - It is easy to enable large amount of IOVA-contiguous memory use-cases
> with IOVA in VA mode.
>
>>
>> It came up in SPDK tests, but it's certainly nothing SPDK-specific, I can't give
>> you the steps but it should be reproducible even with testpmd.
>>
>> The PCI device could have been simply hotplugged to the system after DPDK
>> app start. DPDK didn't know about it at initialization, so it picked
>> RTE_IOVA_VA and then would fail to attach any UIO-bound device ever
>> after:
>>
>> EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
>
> We have RTE_PCI_DRV_NEED_IOVA_AS_VA devices in DPDK, Which can work
> Only on VA. If we default 'PA' incase of DC, then what do with hotplugging on those devices?
>
>
>> EAL: Driver cannot attach the device (0000:00:09.0)
>> EAL: Failed to attach device on primary process
>>
>> UIO is commonly used on systems without IOMMU- including VMs.
>
> The latest machines has IOMMU. Which machines you are testing against,
> Can we detect the machines without IOMMU and switch to PA?
A machine without an IOMMU shouldn't have picked IOVA as VA in the first
place. Perhaps this is something we could fix? I'm not sure how to
detected that condition though, i don't think there's a mechanism to
know that for sure. Some kernels create a "iommu" sysfs directories, but
i'm not too sure if they're 1) there for older kernels we support, and
2) always there.
On machines with IOMMU, VFIO should be the default, and we should
discourage people from using igb_uio. Is there any reason why SPDK is
not using VFIO by default?
On my machine, "/sys/devices/virtual/iommu" exists when IOMMU is
enabled, but doesn't exist if it isn't ("/sys/class/iommu" exists in
both cases, but is empty when IOMMU is disabled). Perhaps we could go
off that?
--
Thanks,
Anatoly
More information about the dev
mailing list