[PATCH 00/10] net/mlx5: improve MAC address and VLAN add latency

[Slava Ovsiienko]

For the entire series:

Acked-by: Viacheslav Ovsiienko <viacheslavo at nvidia.com>

> -----Original Message-----
> From: Dariusz Sosnowski <dsosnowski at nvidia.com>
> Sent: Thursday, October 17, 2024 10:57 AM
> To: Slava Ovsiienko <viacheslavo at nvidia.com>; Bing Zhao
> <bingz at nvidia.com>; Ori Kam <orika at nvidia.com>; Suanming Mou
> <suanmingm at nvidia.com>; Matan Azrad <matan at nvidia.com>
> Cc: dev at dpdk.org
> Subject: [PATCH 00/10] net/mlx5: improve MAC address and VLAN add
> latency
> 
> Whenever a new MAC address is added to the port, mlx5 PMD will:
> 
> - Add this address to `dev->data->mac_addrs[]`.
> - Destroy all control flow rules.
> - Recreate all control flow rules.
> 
> Similar logic is also implemented for VLAN filters.
> 
> Because of such logic, the latency of adding the new MAC address (i.e.,
> latency of `rte_eth_dev_mac_addr_add()` function call) is actually linear to
> number of MAC addresses already configured.
> Since each operation of creating/destroying a control flow rule, involves an
> `ioctl()` syscall, on some setups the latency of adding a single MAC address
> can reach ~100ms, when port is operating with >= 100 MAC addresses.
> The same problem exists for VLAN filters (and even compounded by it).
> 
> This patchset aims to resolve these issues, by reworking how mlx5 PMD
> handles adding/removing MAC addresses and VLAN filters.
> Instead of recreating all control flow rules, only necessary flow rules will be
> created/removed on each operation, thus minimizing number of syscalls
> triggered.
> 
> Summary of patches:
> 
> - Patch 1-2 - Extends existing `mlx5_hw_ctrl_flow_type` enum with special
> variants,
>   which will be used for tracking MAC and VLAN control flow rules.
> - Patch 3-4 - Refactors HWS code for control flow rule creation to allow
>   creation of specific control flow rules with unicast MAC/VLAN match.
>   Also functions are added for deletion of specific rules.
> - Patch 5-6 - Prepares the control flow rules list, used by HWS flow engine,
>   to be used by other flow engine.
>   Goal is to reuse the similar logic in Verbs and DV flow engines.
> - Patch 7-8 - Adjusts legacy flow engines, so that unicast DMAC/VLAN control
> flow rules
>   are added to the control flow rules list.
>   Also exposes functions for creating/destroying specific ones.
> - Patch 9-10 - Extends `mlx5_traffic_*` interface with
> `mlx5_traffic_mac_add/remove` and
>   `mlx5_traffic_vlan_add/remove` functions.
>   They are used in implementations of DPDK APIs for adding/removing MAC
> addresses/VLAN filters
>   and their goal is to update the set of control flow rules in a minimal number
> of steps possible,
>   without recreating the rules.
> 
> As a result of these patches the time to add 128th MAC address, after 127th
> was added drops **from ~72 ms to ~197 us** (at least on my setup).
> 
> Dariusz Sosnowski (10):
>   net/mlx5: track unicast DMAC control flow rules
>   net/mlx5: add checking if unicast flow rule exists
>   net/mlx5: rework creation of unicast flow rules
>   net/mlx5: support destroying unicast flow rules
>   net/mlx5: rename control flow rules types
>   net/mlx5: shared init of control flow rules
>   net/mlx5: add legacy unicast flow rules management
>   net/mlx5: add legacy unicast flow rule registration
>   net/mlx5: add dynamic unicast flow rule management
>   net/mlx5: optimize MAC address and VLAN filter handling
> 
>  drivers/net/mlx5/linux/mlx5_os.c      |   3 +
>  drivers/net/mlx5/meson.build          |   1 +
>  drivers/net/mlx5/mlx5.h               |  62 +++--
>  drivers/net/mlx5/mlx5_flow.c          | 149 ++++++++++-
>  drivers/net/mlx5/mlx5_flow.h          |  36 +++
>  drivers/net/mlx5/mlx5_flow_hw.c       | 349 ++++++++++++++++++++------
>  drivers/net/mlx5/mlx5_flow_hw_stubs.c |  68 +++++
>  drivers/net/mlx5/mlx5_mac.c           |  41 ++-
>  drivers/net/mlx5/mlx5_trigger.c       | 262 ++++++++++++++++++-
>  drivers/net/mlx5/mlx5_vlan.c          |   9 +-
>  drivers/net/mlx5/windows/mlx5_os.c    |   3 +
>  11 files changed, 867 insertions(+), 116 deletions(-)  create mode 100644
> drivers/net/mlx5/mlx5_flow_hw_stubs.c
> 
> --
> 2.39.5