[Patch v4 01/17] net/mana: add basic driver, build environment and doc

[Ferruh Yigit]

On 8/29/2022 8:58 AM, Thomas Monjalon wrote:
> 22/08/2022 20:27, Long Li:
>>> Subject: Re: [Patch v4 01/17] net/mana: add basic driver, build environment
>>> and doc
>>>
>>> On 8/22/2022 4:03 PM, Ferruh Yigit wrote:
>>>>> +                             struct rte_pci_device *pci_dev,
>>>>> +                             struct rte_ether_addr *mac_addr) {
>>>>> +       struct ibv_device **ibv_list;
>>>>> +       int ibv_idx;
>>>>> +       struct ibv_context *ctx;
>>>>> +       struct ibv_device_attr_ex dev_attr;
>>>>> +       int num_devices;
>>>>> +       int ret = 0;
>>>>> +       uint8_t port;
>>>>> +       struct mana_priv *priv = NULL;
>>>>> +       struct rte_eth_dev *eth_dev = NULL;
>>>>> +       bool found_port;
>>>>> +
>>>>> +       ibv_list = ibv_get_device_list(&num_devices);
>>>>> +       for (ibv_idx = 0; ibv_idx < num_devices; ibv_idx++) {
>>>>> +               struct ibv_device *ibdev = ibv_list[ibv_idx];
>>>>> +               struct rte_pci_addr pci_addr;
>>>>> +
>>>>> +               DRV_LOG(INFO, "Probe device name %s dev_name %s
>>>>> ibdev_path %s",
>>>>> +                       ibdev->name, ibdev->dev_name,
>>>>> +ibdev->ibdev_path);
>>>>> +
>>>>> +               if (mana_ibv_device_to_pci_addr(ibdev, &pci_addr))
>>>>> +                       continue;
>>>>> +
>>>>> +               /* Ignore if this IB device is not this PCI device */
>>>>> +               if (pci_dev->addr.domain != pci_addr.domain ||
>>>>> +                   pci_dev->addr.bus != pci_addr.bus ||
>>>>> +                   pci_dev->addr.devid != pci_addr.devid ||
>>>>> +                   pci_dev->addr.function != pci_addr.function)
>>>>> +                       continue;
>>>>> +
>>>>
>>>> As far as I understand, intention of this loop is to find 'ibdev'
>>>> matching this device, code gooes through all "ibv device list" for
>>>> this, I wonder if there is a easy way for doing this, like a sysfs
>>>> entry to help getting this information?
>>>> And how mlx4/5 does this?
>>>
>>> Since there are multiple RDMA devices now, does it make sense to have
>>> RDMA bus driver, which can hide some PCIe details under bus, and driver can
>>> get PCI and ibdev information during probe?
>>
>> Mellanox drivers use a similar way to go through the list of IB devices.
>>
>> Matan, Viacheslav, what are your thoughts on implementing a bus for RDMA devices?
> 
> These are PCI devices first.
> Is it a good idea to have a bus driver on top of another one?
> What would be the responsibility of such a bus driver?
> 

As far as I can see a 'ibdev' device struct is required to be able to 
configure the device, so rdma bus can be on top of PCI and find matching 
'ibdev' device for it, and driver probe can get pci and ibdev 
information directly, instead of each rdma device driver walking through 
all ibdev devices (ibv_get_device_list()).