[dpdk-dev] [PATCH 3/7] eal: move virtual device probing into a bus

[Wiles, Keith]

> On Feb 15, 2017, at 11:25 AM, Jerin Jacob <jerin.jacob at caviumnetworks.com> wrote:
> 
> On Wed, Feb 15, 2017 at 02:27:47PM +0000, Shreyansh Jain wrote:
>>> -----Original Message-----
>>> From: Wiles, Keith [mailto:keith.wiles at intel.com]
>>> Sent: Wednesday, February 15, 2017 7:53 PM
>>> To: Shreyansh Jain <shreyansh.jain at nxp.com>
>>> Cc: Jan Blunck <jblunck at infradead.org>; dev at dpdk.org
>>> Subject: Re: [dpdk-dev] [PATCH 3/7] eal: move virtual device probing into a
>>> bus
>>> 
>>> 
>>>> On Feb 15, 2017, at 8:15 AM, Shreyansh Jain <shreyansh.jain at nxp.com> wrote:
>>>> 
>>>> On Wednesday 15 February 2017 07:41 PM, Shreyansh Jain wrote:
>>>>> On Wednesday 15 February 2017 03:32 PM, Jan Blunck wrote:
>>>>>> This is a refactoring of the virtual device probing which moves into into
>>>>>> a proper bus structure.
>>>>>> 
>>>>>> Signed-off-by: Jan Blunck <jblunck at infradead.org>
>>>>>> ---
>>>>>> lib/librte_eal/common/eal_common_dev.c  | 22 -----------------
>>>>>> lib/librte_eal/common/eal_common_vdev.c | 44
>>>>>> +++++++++++++++++++++++++++++++++
>>>>>> 2 files changed, 44 insertions(+), 22 deletions(-)
>>>>>> 
>>>>> 
>>>>> [...]
>>>>> 
>>>>>> 
>>>>>> diff --git a/lib/librte_eal/common/eal_common_vdev.c
>>>>>> b/lib/librte_eal/common/eal_common_vdev.c
>>>>>> index 7d6e54f..523a3d6 100644
>>>>>> --- a/lib/librte_eal/common/eal_common_vdev.c
>>>>>> +++ b/lib/librte_eal/common/eal_common_vdev.c
>>>>>> @@ -37,8 +37,10 @@
>>>>>> #include <stdint.h>
>>>>>> #include <sys/queue.h>
>>>>>> 
>>>>> [...]
>>>>> 
>>>>>> +
>>>>>> +static struct rte_bus rte_vdev_bus = {
>>>>>> +    .scan = vdev_scan,
>>>>>> +    .probe = vdev_probe,
>>>>>> +};
>>>>>> +
>>>>>> +RTE_REGISTER_BUS_LATE(virtual, rte_vdev_bus);
>>>>>> 
>>>>> 
>>>>> Does it matter if VDEV buses are registered before or after other
>>>>> buses? Either way, the callbacks would be called in the order specified
>>>>> in EAL.
>>>>> 
>>>>> 
>>>> 
>>>> Just ignore this comment - I am misunderstood something.
>>>> 
>>>> But another question: Is there specific reason VDEV should be
>>> registered/scanned *after* other devices? Is there some specific problem if
>>> we do otherwise? (I think this is should be done, but I don't have a specific
>>> reason).
>>> 
>>> Does the bonding driver which uses physical devices need to be registered
>>> after physical ones? In Pktgen I noticed the vdev after the physical ports
>>> and I could not blacklist them as the bonding driver needed them, which
>>> caused the bonding ports to have a greater port number. In the case of pktgen
>>> the bonding ports were up around 8 or 10 and caused the display to not show
>>> the bonding ports. This is really just a usability problem for the developer
>>> using Pktgen. I would like to see the vdev devices first, but as long as the
>>> drivers (like bonding) are fine with them being first.
>> 
>> Ah, now I remember - there was a patch from Jerin for this.
>> Probably he is the best person to comment here.
>> (I don't have much insight here).
> 
> commit f4ce209a8ce5f416b61c76cee773bc54749e2048
> Author: Jerin Jacob <jerin.jacob at caviumnetworks.com>
> Date:   Sun Nov 20 13:30:50 2016 +0530
> 
>    eal: postpone vdev initialization
> 
>    Some platform like octeontx may use pci and
>    vdev based combined device to represent a logical
>    dpdk functional device.In such case, postponing the
>    vdev initialization after pci device
>    initialization will provide the better view of
>    the pci device resources in the system in
>    vdev's probe function, and it allows better
>    functional subsystem registration in vdev probe
>    function.
> 
>    As a bonus, This patch fixes a bond device
>    initialization use case.
> 
>    example command to reproduce the issue:
>    ./testpmd -c 0x2  --vdev 'eth_bond0,mode=0,
>    slave=0000:02:00.0,slave=0000:03:00.0' --
>    --port-topology=chained
> 
>    root cause:
>    In existing case(vdev initialization and then pci
>    initialization), creates three Ethernet ports with
>    following port ids
>    0 - Bond device
>    1 - PCI device 0
>    2 - PCI devive 1
> 
>    Since testpmd, calls the configure/start on all the ports on
>    start up,it will translate to following illegal setup sequence

I guess I see this differently, meaning we modified the system to put vdev devices last only because we do not have clean way to startup the system for pdev/vdev devices. The application should be agnostic to the devices being started and the system needs to determine the correct order without a chicken and egg problem. The test-pmd application just starts from 0 to n to initialize devices, which he should be able to do in any order. It is possible the application could initialize the devices (pdev/vdev) in any order, which the current design would break if they tried to init the bonding driver first.

What happens if a vdev needs to be initialized before a pdev device?

Not saying we need to solve this problem now, but need to figure this out some how. Maybe we need a priority for pdev/vdev devices to determine init order????

> 
>    1)bond device configure/start
>    1.1) pci device0 stop/configure/start
>    1.2) pci device1 stop/configure/start
>    2)pci device 0 configure(illegal setup case,
>    as device in start state)
> 
>    The fix changes the initialization sequence and
>    allow initialization in following valid setup order
>    1) pcie device 0 configure/start
>    2) pcie device 1 configure/start
>    3) bond device 2 configure/start
>    3.1) pcie device 0/stop/configure/start
>    3.2) pcie device 1/stop/configure/start

Regards,
Keith