[RFC PATCH 0/3] flow_compile: textual flow rule compiler

[Stephen Hemminger]

Background
----------

Multiple efforts over the past few cycles have tried to make
testpmd's flow rule grammar reusable from outside testpmd.
External applications that need rte_flow want a documented way
to turn human-written rules into the rte_flow_attr/item/action
arrays accepted by rte_flow_create().

The most recent attempt is Lukas Sismis's series, currently at
v12:

  http://patches.dpdk.org/project/dpdk/list/?series=37384  (or
  most recent thread on dev at dpdk.org)

That series factors testpmd's existing cmdline_flow.c into a
library and updates testpmd to consume it.  It works, but
inherits two properties of cmdline_flow.c that I think are worth
avoiding in a reusable library:

  - Coupling to librte_cmdline.  Even after the v12 split into
    a "simple" part and a "cmdline" part, the parser is still
    organized around testpmd's command interpreter, and v12 has
    cmdline depending on ethdev to break a previous circular
    dependency.  A library used by daemons, control planes, or
    unit tests should not need that.

  - Ad-hoc grammar.  cmdline_flow.c implements parsing per-token
    in long dispatch logic; the grammar emerges from the code
    rather than being stated, and adding a new flow item
    requires touching the parser.

This RFC explores a different shape and is posted to ask the
list which one is preferred before more work goes into either.

So I started a new green field library for parsing flow rules
(with the help of AI assistance). It is only a few hours old,
but passes tests and passes review.

This series
-----------

lib/flow_compile -- a small new library providing the same
service via a pcap_compile()-style API:

    char errbuf[RTE_FLOW_COMPILE_ERRBUF_SIZE];
    struct rte_flow_compile *fc = rte_flow_compile(rule, errbuf);
    if (fc == NULL)
            fail(errbuf);            /* "line:col: message" */

    rte_flow_compile_create(port_id, fc, &flow_error);
    rte_flow_compile_free(fc);

Design properties:

  - Hand-rolled lexer + recursive descent parser.  No flex/bison.
  - Parser is driven entirely by descriptor tables of items and
    actions.  Adding a new flow item is a table edit, not a
    parser change.  A custom-setter hook on each field covers
    the layouts that do not fit a plain byte range (bitfields,
    indirect arrays).
  - Dependencies: rte_ethdev and rte_net only.  No librte_cmdline,
    no flex/bison.  Builds clean on Linux, FreeBSD, and Windows.
  - Per-allocation rte_zmalloc for spec/mask/last/conf payloads;
    rte_flow_compile_free() walks the pattern and action arrays.
    ASan/LSan run clean on the autotest.

The grammar follows testpmd's syntax closely so familiar rules
carry over:

    ingress pattern eth / ipv4 src is 10.0.0.1 / end
    actions queue index 3 / count / end

and is documented as a formal BNF in the programmer's guide
chapter (patch 2).

Initial coverage: eth, vlan, ipv4, ipv6, tcp, udp, vxlan,
port_id, port_representor, represented_port items; drop,
passthru, queue, mark, jump, count, port_id and representor
variants, of_pop_vlan, vxlan_decap actions.  Variable-conf
items and actions (RSS, RAW) require custom setters and are
deferred to a follow-up.

What this RFC is *not*
----------------------

Not a replacement for cmdline_flow.c in testpmd.  If the shape
here is acceptable, the next step is a separate series adding a
"flow compile <port> <rule>" command in testpmd alongside the
existing parser, so users can adopt the library incrementally
without breaking scripts that depend on the current syntax.

What I'd like feedback on
-------------------------

1. API shape.  pcap_compile-style (one string -> opaque object ->
   arrays) versus the three-call attr/pattern/actions form
   Sismis's v12 exposes.  What does your application actually
   want?

2. Library placement.  Stand-alone at lib/flow_compile/ versus
   addition to lib/ethdev.  This series treats it as a
   control-path parser layered on top of ethdev rather than
   part of ethdev itself; v12 places its parser inside ethdev.

3. Table-driven extension model.  Is "to add a new flow item,
   add a row to the descriptor table" the right contract?
   Should the tables live alongside each rte_flow_item_*
   definition in rte_flow.h, or in their own file as here?

4. Convergence.  If this design is preferred, I'm happy to
   coordinate with Lukas to fold in the testpmd-side changes
   from his series.

5. API code. Is it readable enough. Lots of code here
   but doing lexer/parser is prime template territory
   and AI has a good chance here.

Stephen Hemminger (3):
  flow_compile: introduce textual flow rule compiler
  doc: add programmer's guide for flow rule compiler
  test/flow_compile: add unit tests for flow rule compiler

 MAINTAINERS                                    |    8 +
 app/test/meson.build                           |    2 +
 app/test/test_flow_compile.c                   |  230 +++
 doc/guides/prog_guide/flow_compile_lib.rst     |  170 ++
 doc/guides/prog_guide/index.rst                |    1 +
 doc/guides/rel_notes/release_26_07.rst         |    5 +
 lib/flow_compile/flow_compile_lex.c            |  488 +++++
 lib/flow_compile/flow_compile_parse.c          |  634 ++++++
 lib/flow_compile/flow_compile_priv.h           |  181 ++
 lib/flow_compile/flow_compile_tables.c         |  245 +++
 lib/flow_compile/meson.build                   |   15 +
 lib/flow_compile/rte_flow_compile.h            |  158 ++
 lib/flow_compile/rte_flow_compile_api.c        |  132 ++
 lib/flow_compile/version.map                   |   13 +
 lib/meson.build                                |    1 +
 15 files changed, 2283 insertions(+)
 create mode 100644 app/test/test_flow_compile.c
 create mode 100644 doc/guides/prog_guide/flow_compile_lib.rst
 create mode 100644 lib/flow_compile/flow_compile_lex.c
 create mode 100644 lib/flow_compile/flow_compile_parse.c
 create mode 100644 lib/flow_compile/flow_compile_priv.h
 create mode 100644 lib/flow_compile/flow_compile_tables.c
 create mode 100644 lib/flow_compile/meson.build
 create mode 100644 lib/flow_compile/rte_flow_compile.h
 create mode 100644 lib/flow_compile/rte_flow_compile_api.c
 create mode 100644 lib/flow_compile/version.map

-- 
2.43.7


Stephen Hemminger (3):
  flow_compile: introduce textual flow rule compiler
  doc: add programmer's guide for flow rule compiler
  test/flow_compile: add unit tests for flow rule compiler

 MAINTAINERS                                |   7 +
 app/test/meson.build                       |   1 +
 app/test/test_flow_compile.c               | 234 ++++++++
 doc/guides/prog_guide/flow_compile_lib.rst | 272 +++++++++
 doc/guides/prog_guide/index.rst            |   1 +
 doc/guides/rel_notes/release_26_07.rst     |   6 +
 lib/flow_compile/flow_compile_lex.c        | 510 +++++++++++++++++
 lib/flow_compile/flow_compile_parse.c      | 634 +++++++++++++++++++++
 lib/flow_compile/flow_compile_priv.h       | 181 ++++++
 lib/flow_compile/flow_compile_tables.c     | 245 ++++++++
 lib/flow_compile/meson.build               |  15 +
 lib/flow_compile/rte_flow_compile.h        | 158 +++++
 lib/flow_compile/rte_flow_compile_api.c    | 132 +++++
 lib/meson.build                            |   1 +
 14 files changed, 2397 insertions(+)
 create mode 100644 app/test/test_flow_compile.c
 create mode 100644 doc/guides/prog_guide/flow_compile_lib.rst
 create mode 100644 lib/flow_compile/flow_compile_lex.c
 create mode 100644 lib/flow_compile/flow_compile_parse.c
 create mode 100644 lib/flow_compile/flow_compile_priv.h
 create mode 100644 lib/flow_compile/flow_compile_tables.c
 create mode 100644 lib/flow_compile/meson.build
 create mode 100644 lib/flow_compile/rte_flow_compile.h
 create mode 100644 lib/flow_compile/rte_flow_compile_api.c

-- 
2.53.0