[dpdk-dev] [PATCH v3 05/11] bus: get iommu class

santosh santosh.shukla at caviumnetworks.com
Fri Jul 14 12:22:07 CEST 2017


On Friday 14 July 2017 03:09 PM, Hemant Agrawal wrote:

> On 7/14/2017 2:00 PM, santosh wrote:
>> On Friday 14 July 2017 01:37 PM, Hemant Agrawal wrote:
>>
>>> On 7/11/2017 11:46 AM, Santosh Shukla wrote:
>>>> API(rte_bus_get_iommu_class) helps to automatically detect and select
>>>> appropriate iova mapping scheme for iommu capable device on that bus.
>>>>
>>>> Algorithm for iova scheme selection for bus:
>>>> 0. Iterate through bus_list.
>>>> 1. Collect each bus iova mode value and update into 'mode' var.
>>>> 2. Here value '1' is _pa and value '2' is _va mode.
>>>> So mode selection scheme is like:
>>>> if mode == 2 then iova mode is _va.
>>>> if mode == 1 then iova mode is _pa
>>>> if mode  == 3 then iova mode ia _pa.
>>>>
>>>> So mode !=2  will be default iova mode.
>>>>
>>>> Signed-off-by: Santosh Shukla <santosh.shukla at caviumnetworks.com>
>>>> Signed-off-by: Jerin Jacob <jerin.jacob at caviumnetworks.com>
>>>> ---
>>>>  lib/librte_eal/bsdapp/eal/rte_eal_version.map   |  1 +
>>>>  lib/librte_eal/common/eal_common_bus.c          | 23 +++++++++++++++++++++++
>>>>  lib/librte_eal/common/eal_common_pci.c          |  1 +
>>>>  lib/librte_eal/common/include/rte_bus.h         | 22 ++++++++++++++++++++++
>>>>  lib/librte_eal/linuxapp/eal/rte_eal_version.map |  1 +
>>>>  5 files changed, 48 insertions(+)
>>>>
>>>> diff --git a/lib/librte_eal/bsdapp/eal/rte_eal_version.map b/lib/librte_eal/bsdapp/eal/rte_eal_version.map
>>>> index 33c2c32c0..a2dd65a33 100644
>>>> --- a/lib/librte_eal/bsdapp/eal/rte_eal_version.map
>>>> +++ b/lib/librte_eal/bsdapp/eal/rte_eal_version.map
>>>> @@ -202,6 +202,7 @@ DPDK_17.08 {
>>>>      rte_bus_find_by_name;
>>>>      rte_pci_match;
>>>>      rte_pci_get_iommu_class;
>>>> +    rte_bus_get_iommu_class;
>>>>
>>>>  } DPDK_17.05;
>>>>
>>>> diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
>>>> index 08bec2d93..5d5753ac9 100644
>>>> --- a/lib/librte_eal/common/eal_common_bus.c
>>>> +++ b/lib/librte_eal/common/eal_common_bus.c
>>>> @@ -222,3 +222,26 @@ rte_bus_find_by_device_name(const char *str)
>>>>          c[0] = '\0';
>>>>      return rte_bus_find(NULL, bus_can_parse, name);
>>>>  }
>>>> +
>>>> +
>>>> +/*
>>>> + * Get iommu class of devices on the bus.
>>>> + */
>>>> +enum rte_iova_mode
>>>> +rte_bus_get_iommu_class(void)
>>>> +{
>>>> +    int mode = 0;
>>>> +    struct rte_bus *bus;
>>>> +
>>>> +    TAILQ_FOREACH(bus, &rte_bus_list, next) {
>>>> +
>>>> +        if (bus->get_iommu_class)
>>>> +            mode |= bus->get_iommu_class();
>>>> +    }
>>>> +
>>>
>>> If you change the default return as '0' for buses. This code will work.
>>> e.g. PCI will return '0' - when no device is probed. FSL MC will return VA. the default mode will be 'VA'
>>>
>> I'm confused why it won't work for fslmc case?
>>
>> Let me walk through the code:
>>
>> If no-pci device Or (future) no-platform device probed then bus opt
>> to use default mapping scheme .. which is iova_pa(default scheme).
>>
>> Lets take PCI_bus example:
>> bus->get_iommu_class()
>>     ---> bus->_pci_get_iommu_class()
>>         * Now consider that no interface bound to any of PCI device, then
>>           it will return RTE_IOVA_PA mode to rte_bus layer (aka bus->get_iommu_class).
>>           So the iova mapping result from iommu_class scan is RTE_IOVA_PA (default).
>>           It works for PCI_bus case, tested for both iova_va and iova_pa case, no-pci device case.
>>
>> Now in fslmc bus case:
>> bus->get_iommu_class()
>>     ---> bus->_fslmc_get_iommu_class()
>>        
>>         * IIUC your comment - You want fslmc bus to return RTE_IOVA_VA if no device
>>           detected, Right?
> why?
>
As I didn't understood your previous reply:
`e.g. PCI will return '0' - when no device is probed. FSL MC will return VA. the default mode will be 'VA'`

So, I'm asking you that in fslmc bus case - if no device found then are you opting _va scheme or not?
Seems like _not_ per your below comment.
 

> If bus is just present but no device is in use for dpdk, then bus should return 0 and it *should not* participate in the IOMMU class decision.
>
I think, I understand your point..Example if you have no-pci on first PCI bus
but device found on 2nd platform bus then you don't want to fallback to default (/_pa) mode.. 
instead you want to use 2nd bus mode for mapping, which is _va. Right?

If so then In my first version - We did introduced the case called _DC.
_DC:0 --> stands for no-device found case.

> Right now there are only two buses. There can be more buses. (e.g. PCI, platform, fslmc in case of dpaa2 as well).
>
> If the bus is not being used at all, why it influence the decision of other buses.
>
If your referring to above case then I agree, We'll re-introduce _DC state from v1 in next revision.
That will look like
rte_pci_get_iommu_class() {
	int mode = RTE_IOVA_DC; /* '0' */

	return _DC; /* if no device found */
}

Right?

> if no bus has any device, the System default is anyway PA.
>
Right, If no bus present then It's also responsibility of `rte_bus_get_iommu_class`
to use default mapping scheme which is _pa and which It does.

>
>>           if so then your fslmc bus handle should do something like below
>>             -- If no device on fslmc bus : return RTE_IOVA_VA.
>>             -- If device detected on fslmc bus and bound to iommu driver : return RTE_IOVA_VA
>>             -- If device detected fslmc but not bound to iommu drv : return RTE_IOVA_PA..
>>
>> make sense? If not then can you describe fslmc mapping scheme?
>>
>>> if fslmc is not present. The default mode will be PA.
>>>
>>>> +    if (mode != RTE_IOVA_VA) {
>>>> +        /* Use default IOVA mode */
>>>> +        mode = RTE_IOVA_PA;
>>>> +    }
>
> The system default is anyway PA.
>
No, That check is needed for case like 1st bus return with _PA and 2nd bus returns with _VA,
then mode = 3 (Mix mode), which we don't support so (as I mentioned before) its responsibility of
rte_bus_get_iommu_class() to return default mode (_pa). That's why!.

>>>> +    return mode;
>>>> +}
>>>> diff --git a/lib/librte_eal/common/eal_common_pci.c b/lib/librte_eal/common/eal_common_pci.c
>>>> index 8b6ecebd6..bdf2e7c3a 100644
>>>> --- a/lib/librte_eal/common/eal_common_pci.c
>>>> +++ b/lib/librte_eal/common/eal_common_pci.c
>>>> @@ -552,6 +552,7 @@ struct rte_pci_bus rte_pci_bus = {
>>>>          .plug = pci_plug,
>>>>          .unplug = pci_unplug,
>>>>          .parse = pci_parse,
>>>> +        .get_iommu_class = rte_pci_get_iommu_class,
>>>>      },
>>>>      .device_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.device_list),
>>>>      .driver_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.driver_list),
>>>> diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
>>>> index 7a0cfb165..8b2805b7f 100644
>>>> --- a/lib/librte_eal/common/include/rte_bus.h
>>>> +++ b/lib/librte_eal/common/include/rte_bus.h
>>>> @@ -181,6 +181,17 @@ struct rte_bus_conf {
>>>>      enum rte_bus_scan_mode scan_mode; /**< Scan policy. */
>>>>  };
>>>>
>>>> +
>>>> +/**
>>>> + * Get iommu class of devices on the bus.
>>>> + * Check that those devices are attached to iommu driver.
>>>> + *
>>>> + * @return
>>>> + *      enum rte_iova_mode value.
>>>> + */
>>>> +typedef enum rte_iova_mode (*rte_bus_get_iommu_class_t)(void);
>>>> +
>>>> +
>>>>  /**
>>>>   * A structure describing a generic bus.
>>>>   */
>>>> @@ -194,6 +205,7 @@ struct rte_bus {
>>>>      rte_bus_unplug_t unplug;     /**< Remove single device from driver */
>>>>      rte_bus_parse_t parse;       /**< Parse a device name */
>>>>      struct rte_bus_conf conf;    /**< Bus configuration */
>>>> +    rte_bus_get_iommu_class_t get_iommu_class; /**< Get iommu class */
>>>>  };
>>>>
>>>>  /**
>>>> @@ -293,6 +305,16 @@ struct rte_bus *rte_bus_find_by_device(const struct rte_device *dev);
>>>>   */
>>>>  struct rte_bus *rte_bus_find_by_name(const char *busname);
>>>>
>>>> +
>>>> +/**
>>>> + * Get iommu class of devices on the bus.
>>>> + * Check that those devices are attached to iommu driver.
>>>> + *
>>>> + * @return
>>>> + *     enum rte_iova_mode value.
>>>> + */
>>>> +enum rte_iova_mode rte_bus_get_iommu_class(void);
>>>> +
>>>>  /**
>>>>   * Helper for Bus registration.
>>>>   * The constructor has higher priority than PMD constructors.
>>>> diff --git a/lib/librte_eal/linuxapp/eal/rte_eal_version.map b/lib/librte_eal/linuxapp/eal/rte_eal_version.map
>>>> index 044f89c7c..186c7b0fd 100644
>>>> --- a/lib/librte_eal/linuxapp/eal/rte_eal_version.map
>>>> +++ b/lib/librte_eal/linuxapp/eal/rte_eal_version.map
>>>> @@ -207,6 +207,7 @@ DPDK_17.08 {
>>>>      rte_bus_find_by_name;
>>>>      rte_pci_match;
>>>>      rte_pci_get_iommu_class;
>>>> +    rte_bus_get_iommu_class;
>>>>
>>>>  } DPDK_17.05;
>>>>
>>>>
>>>
>>>
>>
>>
>
>



More information about the dev mailing list