[dpdk-dev] [PATCH v3 04/11] linuxapp/eal_pci: get iommu class

santosh santosh.shukla at caviumnetworks.com
Fri Jul 14 11:13:50 CEST 2017


On Friday 14 July 2017 02:16 PM, santosh wrote:

> On Friday 14 July 2017 01:36 PM, Hemant Agrawal wrote:
>
>> On 7/14/2017 1:25 PM, santosh wrote:
>>> On Friday 14 July 2017 01:09 PM, Hemant Agrawal wrote:
>>>
>>>> On 7/11/2017 11:46 AM, Santosh Shukla wrote:
>>>>> Get iommu class of PCI device on the bus and returns preferred iova
>>>>> mapping mode for that bus.
>>>>>
>>>>> Algorithm for iova scheme selection for PCI bus:
>>>>> 0. Look for device attached to vfio kdrv and has .drv_flag set
>>>>> to RTE_PCI_DRV_NEED_IOVA_VA.
>>>>> 1. Look for any device attached to UIO class of driver.
>>>>> 2. Check for vfio-noiommu mode enabled.
>>>>>
>>>>> If 1) & 2) is false and 0) is true then select
>>>>> mapping scheme as iova=va. Otherwise use default
>>>>> mapping scheme (iova_pa).
>>>>>
>>>>> Signed-off-by: Santosh Shukla <santosh.shukla at caviumnetworks.com>
>>>>> Signed-off-by: Jerin Jacob <jerin.jacob at caviumnetworks.com>
>>>>> ---
>>>>> v1 --> v2:
>>>>> - Removed Linux version check in vfio_noiommu func. Refer [1].
>>>>> - Extending autodetction logic for _iommu_class.
>>>>> Refer [2].
>>>>>
>>>>> [1] https://www.mail-archive.com/dev@dpdk.org/msg70108.html
>>>>> [2] https://www.mail-archive.com/dev@dpdk.org/msg70279.html
>>>>>
>>>>>  lib/librte_eal/linuxapp/eal/eal_pci.c           | 66 +++++++++++++++++++++++++
>>>>>  lib/librte_eal/linuxapp/eal/eal_vfio.c          | 19 +++++++
>>>>>  lib/librte_eal/linuxapp/eal/eal_vfio.h          |  4 ++
>>>>>  lib/librte_eal/linuxapp/eal/rte_eal_version.map |  1 +
>>>>>  4 files changed, 90 insertions(+)
>>>>>
>>>>> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
>>>>> index 7d9e1a99b..573caa000 100644
>>>>> --- a/lib/librte_eal/linuxapp/eal/eal_pci.c
>>>>> +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
>>>>> @@ -45,6 +45,7 @@
>>>>>  #include "eal_filesystem.h"
>>>>>  #include "eal_private.h"
>>>>>  #include "eal_pci_init.h"
>>>>> +#include "eal_vfio.h"
>>>>>
>>>>>  /**
>>>>>   * @file
>>>>> @@ -488,6 +489,71 @@ rte_pci_scan(void)
>>>>>      return -1;
>>>>>  }
>>>>>
>>>>> +/*
>>>>> + * Any one of the device bound to uio
>>>>> + */
>>>>> +static inline int
>>>>> +pci_device_bound_uio(void)
>>>>> +{
>>>>> +    struct rte_pci_device *dev = NULL;
>>>>> +
>>>>> +    FOREACH_DEVICE_ON_PCIBUS(dev) {
>>>>> +        if (dev->kdrv == RTE_KDRV_IGB_UIO ||
>>>>> +           dev->kdrv == RTE_KDRV_UIO_GENERIC) {
>>>>> +            return 1;
>>>>> +        }
>>>>> +    }
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Any one of the device has iova as va
>>>>> + */
>>>>> +static inline int
>>>>> +pci_device_has_iova_va(void)
>>>>> +{
>>>>> +    struct rte_pci_device *dev = NULL;
>>>>> +    struct rte_pci_driver *drv = NULL;
>>>>> +
>>>>> +    FOREACH_DRIVER_ON_PCIBUS(drv) {
>>>>> +        if (drv && drv->drv_flags & RTE_PCI_DRV_NEED_IOVA_VA) {
>>>>> +            FOREACH_DEVICE_ON_PCIBUS(dev) {
>>>>> +                if (dev->kdrv == RTE_KDRV_VFIO &&
>>>>> +                    rte_pci_match(drv, dev))
>>>>> +                    return 1;
>>>>> +            }
>>>>> +        }
>>>>> +    }
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Get iommu class of PCI devices on the bus.
>>>>> + */
>>>>> +enum rte_iova_mode
>>>>> +rte_pci_get_iommu_class(void)
>>>>> +{
>>>>> +    bool is_vfio_noiommu_enabled;
>>>>> +    bool has_iova_va;
>>>>> +    bool is_bound_uio;
>>>>> +
>>>>> +    has_iova_va = pci_device_has_iova_va();
>>>>> +    is_bound_uio = pci_device_bound_uio();
>>>>> +    is_vfio_noiommu_enabled = vfio_noiommu_is_enabled() == 1 ? 1 : 0;
>>>>> +
>>>>> +    if (has_iova_va && !is_bound_uio && !is_vfio_noiommu_enabled)
>>>>> +        return RTE_IOVA_VA;
>>>>> +
>>>> PCI is generally present in all platform including dpaa2.
>>>> There may not be any device found or available for dpdk usages in such cases. The PCI bus will still return RTE_IOVA_PA, which will make the system mode as PA.
>>>>
>>> That's the expected behavior. And implementation makes sure
>>> that PCI_bus return default mode aka _PA if no-pci device found.
>>>
>>> Isn't code taking care of same?
>>>
>> I have attached a PCI device to the board. But it is being managed by kernel only.
>>
>> EAL: PCI device 0000:01:00.0 on NUMA socket 0
>> EAL:   probe driver: 8086:10d3 net_e1000_em
>> EAL:   Not managed by a supported kernel driver, skipped
>>
>> So, there are devices in the PCI list. But none of them is probed or being used by dpdk.
>>
>>
> Therefore _pci_get_iommu_class scan result would be _PA, As no device bound to dpdk.
>
>>> Let me walk through the code:
>>>
>>> has_iova_va = 0 (if no pci device then pci_device_has_iov_va() will return 0).
>>>
>>> And if (has_iova_va & ,,,) will fail therefore rte_pci_get_iommu_class() retuns RTE_IOVA_PA mode.
>>> which is default mode. Right?
>>>
>> This will create issue for the 2nd bus, which is a VA bus. The combined mode will becomes '3', so the system mode will be PA.
>>
> Yes, If both modes detected at two different bus 
> then policy is to use default iova mapping mode across the buses(which is _pa).
>
> Are you operating on two different mode like _pa for PCI-bus and _va for fslmc bus in dpaa2? 

Is vfio kernel infrastructure for dpaa2 allows case like below:
0) Use PCI- vfio(/iommu) mode and map vfio.dma_map to RTE_IOVA_PA
AND
1) Use platform/fslmc vfio-platform mode and map vfio.dma_map to RTE_IOVA_VA?

Does dpaa2 supports?

(Speculating) Lets say if dpaa2 platform supports above case 
 then will you see any issue if both buses using default iova_mapping (_pa),
like dpdk pci has currently?



More information about the dev mailing list