[dpdk-dev] [PATCH 2/2] eal: fix IOVA mode selection as VA for pci drivers

[Burakov, Anatoly]

On 12-Jul-19 1:43 PM, Thomas Monjalon wrote:
> 12/07/2019 13:03, Burakov, Anatoly:
>> On 10-Jul-19 10:48 PM, David Marchand wrote:
>>> The incriminated commit broke the use of RTE_PCI_DRV_IOVA_AS_VA which
>>> was intended to mean "driver only supports VA" but had been understood
>>> as "driver supports both PA and VA" by most net drivers and used to let
>>> dpdk processes to run as non root (which do not have access to physical
>>> addresses on recent kernels).
>>>
>>> The check on physical addresses actually closed the gap for those
>>> drivers. We don't need to mark them with RTE_PCI_DRV_IOVA_AS_VA and this
>>> flag can retain its intended meaning.
>>> Document explicitly its meaning.
>>>
>>
>> So, we always assume that all devices support both IOVA as PA and IOVA
>> as VA by default. Well, as long as it's understood and documented :)
> 
> Yes
> Please make sure it is well documented.
> 
>> Unless...
>>
>>
>> <snip>
>>
>>> +
>>> +IOVA Mode is selected by considering what the current usable Devices on the
>>> +system requires and/or supports.
>>> +
>>> +Below is the 2-step heuristic for this choice.
>>> +
>>> +For the first step, EAL asks each bus its requirement in terms of IOVA mode
>>> +and decides on a preferred IOVA mode.
>>> +
>>> +- if all buses report RTE_IOVA_PA, then the preferred IOVA mode is RTE_IOVA_PA,
>>> +- if all buses report RTE_IOVA_VA, then the preferred IOVA mode is RTE_IOVA_VA,
>>> +- if all buses report RTE_IOVA_DC, no bus expressed a preferrence, then the
>>> +  preferred mode is RTE_IOVA_DC,
>>> +- if the buses disagree (at least one wants RTE_IOVA_PA and at least one wants
>>> +  RTE_IOVA_VA), then the preferred IOVA mode is RTE_IOVA_DC (see below with the
>>> +  check on Physical Addresses availability),
>>> +
>>> +The second step is checking if the preferred mode complies with the Physical
>>> +Addresses availability since those are only available to root user in recent
>>> +kernels.
>>> +
>>> +- if the preferred mode is RTE_IOVA_PA but there is no access to Physical
>>> +  Addresses, then EAL init will fail early, since later probing of the devices
>>> +  would fail anyway,
>>> +- if the preferred mode is RTE_IOVA_DC then based on the Physical Addresses
>>> +  availability, the preferred mode is adjusted to RTE_IOVA_PA or RTE_IOVA_VA.
>>> +  In the case when the buses had disagreed on the IOVA Mode at the first step,
>>> +  part of the buses won't work because of this decision.
>>
>> Is there any specific reason why we always prefer PA if physical
>> addresses are available? Since we're already assuming that all devices
>> support PA and VA anyway, what's the harm in enabling VA by default?
> 
> If PA is available, it means we are running as root.
> We can assume that using root is a choice, probably related
> to a preference for PA.
> 
>> I seem to recall there were some concerns around SPDK and PA address
>> availability - doesn't that mean that the assumption regarding PA and VA
>> mode always being supported doesn't actually hold in practice?
>>
>> By the way, the reason i'm harping away on IOVA as VA being the default
>> is because having IOVA as PA is not a free (as in beer) choice - we
>> sacrifice some usability by doing that. Right now, by default, mempool
>> will ask for IOVA-contiguous memory first, and this is slow in IOVA as
>> PA mode - meaning, e.g. testpmd startup time is greatly increased for
>> smaller page sizes because of IOVA as PA mode is the default in DPDK.
>>
>> I would also like to steer people away from using real physical
>> addresses because doing so while requiring lots of IOVA contiguous
>> memory also requires legacy mem mode, which i would rather people not
>> use and grow dependent on, and would like to remove it at some point as
>> it adds a lot of complexity for a corner case.
> 
> That's why we should better encourage to not run as root.
> We need more documentation about how to run as normal user.
> 
>> So, picking address mode is not *just* about whether the device supports
>> them - it has usability implications as well.
> 
> If we consider running as root an exception, then it makes
> sense to pick address mode which fits this exception (PA).
> 

When you put it that way, that does indeed make sense. Typically though, 
developers tend to run as root. I shall hereby stop doing so :)

-- 
Thanks,
Anatoly