[dpdk-dev] [PATCH 1/2] ethdev: add flow action type update as an offload

[Zhang, Qi Z]

> -----Original Message-----
> From: Ori Kam <orika at mellanox.com>
> Sent: Friday, November 8, 2019 8:43 PM
> To: Andrew Rybchenko <arybchenko at solarflare.com>; Zhang, Qi Z
> <qi.z.zhang at intel.com>; Thomas Monjalon <thomas at monjalon.net>
> Cc: dev at dpdk.org; pbhagavatula at marvell.com; Yigit, Ferruh
> <ferruh.yigit at intel.com>; jerinj at marvell.com; Mcnamara, John
> <john.mcnamara at intel.com>; Kovacevic, Marko
> <marko.kovacevic at intel.com>; Adrien Mazarguil
> <adrien.mazarguil at 6wind.com>; david.marchand at redhat.com;
> ktraynor at redhat.com; Olivier Matz <olivier.matz at 6wind.com>
> Subject: RE: [dpdk-dev] [PATCH 1/2] ethdev: add flow action type update as an
> offload
> 
> Yes missed somehow 6.
> One more general comment, what happened until now? The Mark is already
> implemented and working more then 2 years.

Actually we get many complain on i40e driver does not support MARK due to vector path is selected by default
The driver always need a hint to select the mark aware data path by "--pkt-filter-mode =perfect" which is just a workaround.
though this issue has been finally solved since software mark on vPMD is enabled recently on i40e.
but For the new ice driver, we didn't get chance to enable this on vPMD yet, then we meet the same issue.

I know option B can always work, driver can always decide the function call at runtime, but that need a lot rework and may impact performance, while option A just give more flexibility,
and it does not block a driver to enable all the best thing required by option B.

Regards
Qi


> 
> > -----Original Message-----
> > From: Andrew Rybchenko <arybchenko at solarflare.com>
> > Sent: Friday, November 8, 2019 2:20 PM
> > To: Ori Kam <orika at mellanox.com>; Zhang, Qi Z <qi.z.zhang at intel.com>;
> > Thomas Monjalon <thomas at monjalon.net>
> > Cc: dev at dpdk.org; pbhagavatula at marvell.com; Yigit, Ferruh
> > <ferruh.yigit at intel.com>; jerinj at marvell.com; Mcnamara, John
> > <john.mcnamara at intel.com>; Kovacevic, Marko
> > <marko.kovacevic at intel.com>; Adrien Mazarguil
> > <adrien.mazarguil at 6wind.com>; david.marchand at redhat.com;
> > ktraynor at redhat.com; Olivier Matz <olivier.matz at 6wind.com>
> > Subject: Re: [dpdk-dev] [PATCH 1/2] ethdev: add flow action type
> > update as an offload
> >
> > On 11/8/19 3:12 PM, Ori Kam wrote:
> > >
> > > Hi Andrew,
> > >
> > > Thanks very much for the summery, I think this makes the discussion
> > > much
> > better.
> > > (we even got some new folks 😊)
> > >
> > > I vote for B and C,
> > >
> > > Please read my comments below.
> > >
> > > Ori
> > >
> > >> -----Original Message-----
> > >> From: dev <dev-bounces at dpdk.org> On Behalf Of Zhang, Qi Z
> > >> Sent: Friday, November 8, 2019 1:41 PM
> > >> To: Thomas Monjalon <thomas at monjalon.net>; Andrew Rybchenko
> > >> <arybchenko at solarflare.com>
> > >> Cc: Ori Kam <orika at mellanox.com>; dev at dpdk.org;
> > >> pbhagavatula at marvell.com; Yigit, Ferruh <ferruh.yigit at intel.com>;
> > >> jerinj at marvell.com; Mcnamara, John <john.mcnamara at intel.com>;
> > >> Kovacevic, Marko <marko.kovacevic at intel.com>; Adrien Mazarguil
> > >> <adrien.mazarguil at 6wind.com>; david.marchand at redhat.com;
> > >> ktraynor at redhat.com; Olivier Matz <olivier.matz at 6wind.com>
> > >> Subject: Re: [dpdk-dev] [PATCH 1/2] ethdev: add flow action type
> > >> update
> > as
> > >> an offload
> > >>
> > >>
> > >>
> > >>> -----Original Message-----
> > >>> From: dev <dev-bounces at dpdk.org> On Behalf Of Thomas Monjalon
> > >>> Sent: Friday, November 8, 2019 7:04 PM
> > >>> To: Andrew Rybchenko <arybchenko at solarflare.com>
> > >>> Cc: Ori Kam <orika at mellanox.com>; dev at dpdk.org;
> > >>> pbhagavatula at marvell.com; Yigit, Ferruh <ferruh.yigit at intel.com>;
> > >>> jerinj at marvell.com; Mcnamara, John <john.mcnamara at intel.com>;
> > >>> Kovacevic, Marko <marko.kovacevic at intel.com>; Adrien Mazarguil
> > >>> <adrien.mazarguil at 6wind.com>; david.marchand at redhat.com;
> > >>> ktraynor at redhat.com; Olivier Matz <olivier.matz at 6wind.com>
> > >>> Subject: Re: [dpdk-dev] [PATCH 1/2] ethdev: add flow action type
> > update as
> > >> an
> > >>> offload
> > >>>
> > >>> 08/11/2019 11:42, Andrew Rybchenko:
> > >>>> On 11/8/19 1:28 PM, Thomas Monjalon wrote:
> > >>>>> 08/11/2019 09:35, Andrew Rybchenko:
> > >>>>>> The problem:
> > >>>>>> ~~~~~~~~~~~~
> > >>>>>> PMD wants to know before port start if application wants to to
> > >>>>>> use flow MARK/FLAG in the future. It is required because:
> > >>>>>>
> > >>>>>> 1. HW may be configured in a different way to reserve resources
> > >>>>>>    for MARK/FLAG delivery
> > >>>>>>
> > >>>>>> 2. Datapath implementation choice may depend on it (e.g. vPMD
> > >>>>>>    is faster, but does not support MARK)
> > >>>>>
> > >>>>> Thank you for the clear problem statement.
> > >>>>> I agree with it. This is a real design issue.
> > >>>>>
> > >>>>>
> > >>>>>> Discussed solutions:
> > >>>>>> ~~~~~~~~~~~~~~~~~~~~
> > >>>>
> > >>>> May be it is not 100% clear since below are alternatives.
> > >>>>
> > >>>>>> A. Explicit Rx offload suggested by the patch.
> > >>>>>>
> > >>>>>> B. Implicit by validation of a flow rule with MARK/FLAG actions used.
> > >>>>>>
> > >
> > > I don’t agree that this method is implicit it is quit explicit, do
> > > we support it in
> > PMD
> > > with the use case or not. Sound very explicit. You can argue that
> > > the PMD
> > uses
> > > validation flow to understand that it need to support MARK, but from
> > every perspective
> > > it is more correct . what does it mean if the application enabled
> > > mark and
> > never download a flow?
> >
> > It is implicit from PMD point of view, since there is no command to
> > enable it. The decision to enable is the result of seen attempt to
> > validate mark. If few rules are validated, but an application decides
> > to have it disabled finally, how to achieve it?
> > Last validation attempt wins?
> > Other points are covered by (6) below.
> >
> > >>>>>> C. Use dynamic field/flag (i.e. application registers dynamic field
> > >>>>>>    and/or flag and PMD uses lookup to solve the problem) plus part
> > >>>>>>    of (B) to discover if a feature is supported.
> > >>>>>
> > >>>>> The dynamic field should be registered via a new API function
> > >>>>> named '<feature>_init'.
> > >>>>> It means the application must explicit request the feature.
> > >>>>> I agree this is the way to go.
> > >>>>
> > >>>> If I understand your statement correctly, but (C) is not ideal
> > >>>> since it looks global. If registered dynamic field of mbuf and is
> > >>>> flag that the feature should be enabled, it is a flag to all ports/devices.
> > >>>>
> > >>>>>> All solutions require changes in applications which use these
> > >>>>>> features. There is a deprecation notice in place which
> > >>>>>> advertises DEV_RX_OFFLOAD_FLOW_MARK addition, but may be it is
> > >>>>>> OK to substitute it with solution (B) or (C). Solution (C)
> > >>>>>> requires changes since it should be combined with (B) in order
> > >>>>>> to understand if the feature is supported.
> > >>>>>
> > >>>>> I don't understand.
> > >>>>> Application request and PMD support are two different things.
> > >>>>> PMD support must be via rte_flow validation on a case by case
> > anyway.
> > >>>>
> > >>>> I mean that application wants to understand if the feature is
> > >>>> supported. Then, it wants to enable it. In the case of (B), if I
> > >>>> understand the solution correctly, there is no explicit way to
> > >>>> enable, PMD just detects it because of discovery is done (that's
> > >>>> what I mean by "implicit" and it is a drawback from my point of
> > >>>> view, but still could be considered). (C) solves the problem of (B).
> > >>>>
> > >>>>>> Advantages and drawbacks of solutions:
> > >>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > >>>>>> 1. The main drawback of (A) is a "duplication" since we already
> > >>>>>>    have a way to request flow MARK using rte_flow API.
> > >>>>>>    I don't fully agree that it is a duplication, but I agree
> > >>>>>>    that it sounds like duplication and complicates a bit flow
> > >>>>>>    MARK usage by applications. (B) complicates it as well.
> > >>>>>>
> > >>>>>> 2. One more drawback of the solution (A) is the necessity of
> > >>>>>>    similar solution for META and it eats one more offload bit.
> > >>>>>>    Yes, that's true and I think it is not a problem.
> > >>>>>>    It would make it easier for applications to find out if
> > >>>>>>    either MARK or META is supported.
> > >>>>>>
> > >>>>>> 3. The main advantage of the solution (A) is simplicity.
> > >>>>>>    It is simple for application to understand if it supported.
> > >>>>>>    It is simple in PMD to understand that it is required.
> > >>>>>>    It is simple to disable it - just reconfigure.
> > >>>>>>    Also it is easier to document it - just mention that
> > >>>>>>    the offload should be supported and enabled.
> > >>>>>>
> > >
> > > This is not true, for example the PMD support Mark only in Nic mode
> > > While the application needs E-Switch then the PMD reports incorrectly.
> > > Or for example if the mark is supported only as only action, and the
> > application needs
> > > decap and mark. this means that the user will never be able to use
> > > the
> > mark
> > > so the PMD uses slower Rx function without the mark ever being used.
> >
> > It is covered by (6) below.
> >
> > >>>>>> 4. The main advantage of the solution (B) is no "duplication".
> > >>>>>>    I agree that it is valid argument. Solving the problem
> > >>>>>>    without extra entities is always nice, but unfortunately
> > >>>>>>    it is too complex in this case.
> > >>>>>>
> > >
> > > You forgot a one more very important reason, the fact that the mark
> > > may be supported only on some flows. So if the user needs mark with
> > decap
> > > and this combination is not supported the user will never we able to
> > > use
> > mark
> > > while using mark Rx function. So you get slow data path with no
> > > ability to
> > use it.
> >
> > Again, see (6) below.
> >
> > >>>>>> 5. The main drawback of the solution (B) is the complexity.
> > >>>>>>    It is necessary to choose a flow rule which should be used
> > >>>>>>    as a criteria. It could be hardware dependent.
> > >>>>>>    Complex logic is require in PMD if it wants to address the
> > >>>>>>    problem and control MARK delivery based on validated flow
> > >>>>>>    rules. It adds dependency between start/stop processing and
> > >>>>>>    flow rules validation code.
> > >>>>>>    It is pretty complicated to document it.
> > >>>>>>
> > >>>>>> 6. Useless enabling of the offload in the case of solution (A)
> > >>>>>>    if really used flow rules do not support MARK looks like
> > >>>>>>    drawback as well, but easily mitigated by a combination
> > >>>>>>    with solution (B) and only required if the application wants
> > >>>>>>    to dive in the level of optimization and complexity and
> > >>>>>>    makes sense if application knows required flow rules in
> > >>>>>>    advance. So, it is not a problem in this case.
> > >>>>>>
> > >>>>>> 7. Solution (C) has drawbacks of the solution (B) for
> > >>>>>>    applications to understand if these features are supported,
> > >>>>>>    but no drawbacks in PMD, since explicit criteria is used to
> > >>>>>>    enable/disable (dynamic field/flag lookup).
> > >>>>>>
> > >>>>>> 8. Solution (C) is nice since it avoids "duplication".
> > >>>>>>
> > >>>>>> 9. The main drawback of the solution (C) is asymmetry.
> > >>>>>>    As it was discussed in the case of RX_TIMESTAMP
> > >>>>>>    (if I remember it correctly):
> > >>>>>>     - PMD advertises RX_TIMESTAMP offload capability
> > >>>>>>     - application enables the offload
> > >>>>>>     - PMD registers dynamic field for timestamp
> > >>>>>>    Solution (C):
> > >>>>>>      - PMD advertises nothing
> > >>>>>>      - application uses solution (B) to understand if
> > >>>>>>        these features are supported
> > >>>>>>      - application registers dynamic field/flag
> > >>>>>>      - PMD does lookup and solve the problem
> > >>>>>>    The asymmetry could be partially mitigated if RX_TIMESTAMP
> > >>>>>>    solution is changed to require an application to register
> > >>>>>>    dynamic fields and PMD to do lookup if the offload is
> > >>>>>>    enabled. So, the only difference will be in no offload
> > >>>>>>    in the case of flow MARK/FLAG and usage of complex logic
> > >>>>>>    to understand if it is supported or no.
> > >>>>>>    May be it would be really good since it will allow to
> > >>>>>>    have dynamic fields registered before mempool population.
> > >>>>>>
> > >>>>>> 10. Common drawback of solutions (B) and (C) is no granularity.
> > >>>>>>     Solution (A) may be per queue while (B) and (C) cannot be
> > >>>>>>     per queue. Moreover (C) looks global - for all devices.
> > >>>>>>     It could be really painful.
> > >>>>>>
> > >>>>>> (C) is nice, but I still vote for simplicity and granularity of
> > >>>>>> (A).
> > >>>>>
> > >>>>> I vote for clear separation of application needs and PMD
> > >>>>> support, by using the method C (dynamic fields).
> > >>>>> I agree timestamp must use the same path.
> > >>>>> I agree it's complicate because we don't know in advance whether
> > >>>>> a flow rule will be accepted, but that's the reality, config is complex.
> > >>>>
> > >>>> Do you think that global nature of the (C) is acceptable?
> > >>>
> > >>> That's a good question.
> > >>> Maybe the feature request should be per port.
> > >>> In this case, we are back to solution A with a flag per port?
> > >>>
> > >>> Note that A and C will not guarantee that the offload will be possible.
> > >>> We need B (flow rule validation) anyway.
> > >>
> > >> I may not understand how solution B can works well for all the cases.
> > >>
> > >
> > > That is the point of solution B it always work, it is a bit harder
> > > to use maybe, but you can check and decide everything.
> > >
> > >> A rte_flow rule can be issued after dev_start, which means the
> > >> rx_burst function is already selected at that time, so does that
> > >> mean the driver need to switch from a non- mark offload
> > aware
> > >> path to a mark offload aware path without stop device? or it has to
> > >> reject
> > the
> > >> flow?
> > >> The question is if we have 2 data path, one support some offload ,
> > >> one
> > not
> > >> but more fast, which one should be selected during dev_start? Isn't
> > Offload
> > >> widely used to solve this problem?
> > >>
> > >> I think the option A solve all the problems, option C might also
> > >> works, but
> > A is
> > >> looks much straightforward for me.
> > >>
> > >
> > > Solution A my result in selecting incorrect datapath  please see my
> > > answer
> > above.
> > >
> > > As you can guess I'm in favor of B and C (C need B)
> > >
> > >> Regards
> > >> Qi
> > >>
> > >>
> > >>>
> > >>> It seems A, B, C are not alternatives but all required as pieces
> > >>> of a
> > puzzle...
> > >>>
> > >