[dpdk-dev] [PATCH v5 7/9] devarg: change representor ID to bitmap

[Andrew Rybchenko]

On 1/19/21 2:04 PM, Xueming(Steven) Li wrote:
> 
>> -----Original Message-----
>> From: Andrew Rybchenko <andrew.rybchenko at oktetlabs.ru>
>> Sent: Tuesday, January 19, 2021 4:21 PM
>> To: Xueming(Steven) Li <xuemingl at nvidia.com>
>> Cc: dev at dpdk.org; Slava Ovsiienko <viacheslavo at nvidia.com>; Asaf Penso
>> <asafp at nvidia.com>; Ajit Khaparde <ajit.khaparde at broadcom.com>;
>> Somnath Kotur <somnath.kotur at broadcom.com>; John Daley
>> <johndale at cisco.com>; Hyong Youb Kim <hyonkim at cisco.com>; Beilei Xing
>> <beilei.xing at intel.com>; Jeff Guo <jia.guo at intel.com>; Haiyue Wang
>> <haiyue.wang at intel.com>; Matan Azrad <matan at nvidia.com>; Shahaf Shuler
>> <shahafs at nvidia.com>; NBU-Contact-Thomas Monjalon
>> <thomas at monjalon.net>; Ferruh Yigit <ferruh.yigit at intel.com>; Ray Kinsella
>> <mdr at ashroe.eu>; Neil Horman <nhorman at tuxdriver.com>
>> Subject: Re: [PATCH v5 7/9] devarg: change representor ID to bitmap
>>
>> On 1/19/21 10:14 AM, Xueming Li wrote:
>>> The NIC can have multiple PCIe links and can be attached to multiple
>>> hosts, for example the same single NIC can be shared for multiple
>>> server units in the rack. On each PCIe link NIC can provide multiple
>>> PFs and VFs/SFs based on these ones. The full representor identifier
>>> consists of three indices - controller index, PF index, and VF or SF index (if
>> any).
>>>
>>> SR-IOV and SubFunction are created on top of PF. PF index is
>>> introduced because there might be multiple PFs in the bonding
>>> configuration and only bonding device is probed.
>>>
>>> In eth representor comparator callback, ethdev was compared with devarg.
>>> Since ethdev representor port didn't contain controller index and PF
>>> index information, callback returned representor from other PF or
>>> controller.
>>>
>>> This patch changes representor ID to bitmap so that the ethdev
>>> representor comparer callback returns correct ethdev by comparing full
>>> representor information including: controller index, PF index,
>>> representor type, SF or VF index.
>>>
>>> Representor ID bitmap definition:
>>>  xxxx xxxx xxxx xxxx
>>>  |||| |LLL LLLL LLLL vf/sf id
>>>  |||| L 1:sf, 0:vf
>>>  ||LL pf id
>>>  LL controller(host) id
>>
>> What about PF representor case? I.e. representor for entire PF.
>>
>> Also it implies that controller ID 0 is the caller. I.e.
>> special meaning. So, space for just 3 specific contoller left
>>
>> Similar for PF. In fact it is worse. E.g. PMD is bound to the second PF (PF
>> number 1). If so, vf0 means the first VF of the PF itself, i.e. PF 1 VF 0. But,
>> pf0vf0 should mean PF 1 VF 1.
> 
> Agree, need to extend bits width in LTS release.

See my reply to cover mail.

> PF representor is not considered here, how about moving one bit from vf/sf id?
> 1k SF devices should be fine for me so far.

We could reserve max VF/SF number to denote PF itself.

> The controller ID and PF ID is related to the context device, how device configured
> and bonding state is critical for PMD to interpret the IDs. For example:
> "<BDF>,representor=pf1vf1" is valid for bonding device, invalid for standalone device.

I guess it is mlx5 specific. IMHO, pf1vf1 makes sense even
without bonding.

> "c#" is meaningful for multi-host scenario, invalid for normal NIC. PMD is responsible to
> encode representor ID correctly according to device configuration to make Device+ReprID
> unique, because the ReprID is used in device iterator. So the user app should specify
> representor syntax with necessary parts to cover device configuration, PMD should 
> extract required info according to device state.

See cover mail reply.