[dpdk-dev] [RFC 00/21] mlx5: sharing global MR cache between drivers

Michael Baum michaelba at nvidia.com
Tue Aug 17 15:44:20 CEST 2021


There are 5 classes of mlx5 drivers (net\eth, RegEx, vDPA, compress and
crypto). The various drivers are registered under the common mlx5 driver
and are managed by it.
The common driver probing calls in a loop to the probe function of each
driver registered to it.
Each driver creates for itself all the objects required for
communication with the hardware and a global MR cache that manages
memory mappings.
The management of the caches separately by the different drivers is not
very efficient. In fact, the same memory is mapped multiple times to the
HW when more than 1 class use the device.
This feature will move management to the common driver in two phases.

Phase 1: sharing HW objects between drivers on the same device
The communication with the hardware - for any MR handle - is conducted
by the Protection Domain, so we are motivated to share it between the
drivers. However, to create it, we need to give the context of the
device, so the context must also be shared between the drivers.
At this point, we will share the next trio between the drivers (CTX, PD,
pdn) to create an infrastructure that will allow sharing of dependent
objects, particularly the global MR cache.
The common driver itself will create this trio individually for all
drivers before calling their probe function. Then, as a parameter to the
probe function, it will give them a pointer to the structure containing
the trio.

Phase 2: sharing global MR cache between drivers on the same device
The common driver will add to the structure containing the trio and the
structure that manages the global MR cache and keep a list of such
structures for memory management. In each driver, each queue will manage
its own local MR cache. If the queue does not find its cache, it will
search the global MR cache shared by all. Caching access will be through
the pointer that the driver received as a parameter in probing.


Michael Baum (21):
  net/mlx5: fix shared device context creation error flow
  net/mlx5: fix PCI probing error flow
  common/mlx5: add context device structure
  compress/mlx5: use context device structure
  crypto/mlx5: use context device structure
  regex/mlx5: use context device structure
  net/mlx5: improve probe function on Windows
  net/mlx5: improve probe function on Linux
  net/mlx5: improve spawn function
  net/mlx5: use context device structure
  net/mlx5: move NUMA node field to context device
  common/mlx5: add ROCE disable in context device creation
  vdpa/mlx5: use context device structure
  mlx5: update device sent to probing
  mlx5: share context device structure between drivers
  common/mlx5: add HCA attributes to context device structure
  regex/mlx5: use HCA attributes from context device
  vdpa/mlx5: use HCA attributes from context device
  compress/mlx5: use HCA attributes from context device
  crypto/mlx5: use HCA attributes from context device
  net/mlx5: use HCA attributes from context device

 drivers/common/mlx5/linux/mlx5_common_os.c   | 268 ++++++++-
 drivers/common/mlx5/mlx5_common.c            | 273 +++++++++-
 drivers/common/mlx5/mlx5_common.h            |  35 +-
 drivers/common/mlx5/mlx5_common_private.h    |   6 -
 drivers/common/mlx5/version.map              |   2 +
 drivers/common/mlx5/windows/mlx5_common_os.c | 207 ++++++-
 drivers/compress/mlx5/mlx5_compress.c        | 112 +---
 drivers/crypto/mlx5/mlx5_crypto.c            | 111 +---
 drivers/crypto/mlx5/mlx5_crypto.h            |   4 +-
 drivers/crypto/mlx5/mlx5_crypto_dek.c        |   5 +-
 drivers/net/mlx5/linux/mlx5_ethdev_os.c      |   8 +-
 drivers/net/mlx5/linux/mlx5_mp_os.c          |   9 +-
 drivers/net/mlx5/linux/mlx5_os.c             | 543 +++++++++----------
 drivers/net/mlx5/linux/mlx5_verbs.c          |  55 +-
 drivers/net/mlx5/mlx5.c                      |  85 ++-
 drivers/net/mlx5/mlx5.h                      |  17 +-
 drivers/net/mlx5/mlx5_devx.c                 |  35 +-
 drivers/net/mlx5/mlx5_flow.c                 |   6 +-
 drivers/net/mlx5/mlx5_flow_aso.c             |  24 +-
 drivers/net/mlx5/mlx5_flow_dv.c              |  51 +-
 drivers/net/mlx5/mlx5_flow_verbs.c           |   4 +-
 drivers/net/mlx5/mlx5_mr.c                   |  14 +-
 drivers/net/mlx5/mlx5_txpp.c                 |  27 +-
 drivers/net/mlx5/windows/mlx5_ethdev_os.c    |  14 +-
 drivers/net/mlx5/windows/mlx5_os.c           | 285 ++--------
 drivers/regex/mlx5/mlx5_regex.c              |  74 +--
 drivers/regex/mlx5/mlx5_regex.h              |  23 +-
 drivers/regex/mlx5/mlx5_regex_control.c      |  12 +-
 drivers/regex/mlx5/mlx5_regex_fastpath.c     |  18 +-
 drivers/regex/mlx5/mlx5_rxp.c                |  64 ++-
 drivers/vdpa/mlx5/mlx5_vdpa.c                | 210 +------
 drivers/vdpa/mlx5/mlx5_vdpa.h                |   4 +-
 drivers/vdpa/mlx5/mlx5_vdpa_event.c          |  19 +-
 drivers/vdpa/mlx5/mlx5_vdpa_lm.c             |   6 +-
 drivers/vdpa/mlx5/mlx5_vdpa_mem.c            |  13 +-
 drivers/vdpa/mlx5/mlx5_vdpa_steer.c          |  10 +-
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c          |  16 +-
 37 files changed, 1414 insertions(+), 1255 deletions(-)

-- 
2.25.1



More information about the dev mailing list