[dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers
michaelba at nvidia.com
michaelba at nvidia.com
Thu Sep 30 19:28:04 CEST 2021
From: Michael Baum <michaelba at oss.nvidia.com>
The MLNX PMD supports 5 classes (net, RegEx, vDPA, Compress and Crypto).
The various drivers are registered under the common driver, and managed
by it.
In the common driver probing, it calls in a loop the probe function of
each driver registered to it. Each driver creates for itself all the
objects required for communication with the hardware, as well as a
global MR cache that manages memory mappings.
The management of the caches separately by the different drivers, is not
very efficient. In fact the same memory is managed several times and
each caching handle is done by doubling the number of drivers, so we
want to manage this cache jointly by the common driver.
This feature will move management to common driver in two phases.
Phase 1: sharing HW objects between drivers (11 patches)
The communication with the hardware - for any MR handle - is conducted
by the Protection Domain, so we are motivated to share it between the
drivers. However, to create it we need to give the context of the
device, so the context must also be shared between the drivers.
At this point we will share between the drivers the next trio (CTX, PD,
pdn) to create an infrastructure that will allow sharing of dependent
objects, and in particular the global MR cache.
The common driver itself will create this trio individually for all
drivers, before calling their probe function. As a parameter to the
probe function, it will give them a pointer to the structure containing
the trio.
Phase 2: sharing global MR cache between drivers (7 patches)
The common driver will add to the structure containing the trio also the
structure that manages the global MR cache, and will keep a list of such
structures for memory management. In each driver, each queue will manage
its own local MR cache. If the queue does not find its cache, it will
switch to searching the global MR cache shared by all. Caching access
will be through the pointer that the driver received as a parameter in
probing.
Depends-on: series-19267 ("mempool: add event callbacks")
https://patchwork.dpdk.org/project/dpdk/list/?series=19267
Depends-on: series-19097 ("net/mlx5: support new global device syntax")
https://patchwork.dpdk.org/project/dpdk/list/?series=19097
Michael Baum (18):
net/mlx5/windows: fix miss callback register for mem event
common/mlx5: share basic probing with the internal drivers
common/mlx5: share common definitions
common/mlx5: share memory related devargs
net/mlx5/windows: rearrange probing code
common/mlx5: move basic probing functions to common
net/mlx5: remove redundant flag in device config
common/mlx5: share device context object
common/mlx5: add ROCE disable in context device creation
common/mlx5: share the protection domain object
common/mlx5: share the HCA capabilities handle
net/mlx5: remove redundancy in MR file
common/mlx5: add MR ctrl init function
common/mlx5: add global MR cache create function
common/mlx5: share MR top-half search function
common/mlx5: share MR management
common/mlx5: support device DMA map and unmap
common/mlx5: share MR mempool registration
drivers/common/mlx5/linux/mlx5_common_os.c | 280 +++++++-
drivers/common/mlx5/linux/mlx5_common_os.h | 16 +-
drivers/common/mlx5/linux/mlx5_common_verbs.c | 96 ++-
drivers/common/mlx5/linux/mlx5_nl.c | 2 +-
drivers/common/mlx5/linux/mlx5_nl.h | 6 +-
drivers/common/mlx5/mlx5_common.c | 596 +++++++++++++++---
drivers/common/mlx5/mlx5_common.h | 96 +--
drivers/common/mlx5/mlx5_common_defs.h | 45 ++
drivers/common/mlx5/mlx5_common_mp.h | 11 +
drivers/common/mlx5/mlx5_common_mr.c | 192 +++++-
drivers/common/mlx5/mlx5_common_mr.h | 76 ++-
drivers/common/mlx5/mlx5_common_private.h | 6 -
drivers/common/mlx5/mlx5_devx_cmds.h | 12 +-
drivers/common/mlx5/mlx5_malloc.h | 1 -
drivers/common/mlx5/version.map | 29 +-
drivers/common/mlx5/windows/mlx5_common_os.c | 245 ++++++-
drivers/common/mlx5/windows/mlx5_common_os.h | 9 +-
drivers/compress/mlx5/mlx5_compress.c | 223 ++-----
drivers/crypto/mlx5/mlx5_crypto.c | 201 +-----
drivers/crypto/mlx5/mlx5_crypto.h | 6 +-
drivers/crypto/mlx5/mlx5_crypto_dek.c | 5 +-
drivers/net/mlx5/linux/mlx5_ethdev_os.c | 6 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 12 +-
drivers/net/mlx5/linux/mlx5_os.c | 511 ++++-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 99 +--
drivers/net/mlx5/linux/mlx5_verbs.h | 2 -
drivers/net/mlx5/meson.build | 1 -
drivers/net/mlx5/mlx5.c | 241 ++-----
drivers/net/mlx5/mlx5.h | 57 +-
drivers/net/mlx5/mlx5_defs.h | 22 +-
drivers/net/mlx5/mlx5_devx.c | 39 +-
drivers/net/mlx5/mlx5_flow.c | 6 +-
drivers/net/mlx5/mlx5_flow_aso.c | 50 +-
drivers/net/mlx5/mlx5_flow_dv.c | 66 +-
drivers/net/mlx5/mlx5_flow_verbs.c | 4 +-
drivers/net/mlx5/mlx5_mr.c | 397 ------------
drivers/net/mlx5/mlx5_mr.h | 26 -
drivers/net/mlx5/mlx5_rx.c | 16 +-
drivers/net/mlx5/mlx5_rx.h | 20 +-
drivers/net/mlx5/mlx5_rxq.c | 11 +-
drivers/net/mlx5/mlx5_rxtx.c | 1 -
drivers/net/mlx5/mlx5_rxtx.h | 27 -
drivers/net/mlx5/mlx5_rxtx_vec.h | 1 -
drivers/net/mlx5/mlx5_trigger.c | 8 +-
drivers/net/mlx5/mlx5_tx.c | 1 -
drivers/net/mlx5/mlx5_tx.h | 30 +-
drivers/net/mlx5/mlx5_txpp.c | 23 +-
drivers/net/mlx5/mlx5_txq.c | 10 +-
drivers/net/mlx5/windows/mlx5_ethdev_os.c | 12 +-
drivers/net/mlx5/windows/mlx5_os.c | 402 ++----------
drivers/regex/mlx5/mlx5_regex.c | 138 +---
drivers/regex/mlx5/mlx5_regex.h | 27 +-
drivers/regex/mlx5/mlx5_regex_control.c | 19 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 47 +-
drivers/regex/mlx5/mlx5_rxp.c | 62 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 212 +------
drivers/vdpa/mlx5/mlx5_vdpa.h | 5 +-
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 25 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 6 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 13 +-
drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 11 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 15 +-
62 files changed, 2065 insertions(+), 2769 deletions(-)
create mode 100644 drivers/common/mlx5/mlx5_common_defs.h
delete mode 100644 drivers/net/mlx5/mlx5_mr.c
delete mode 100644 drivers/net/mlx5/mlx5_mr.h
--
2.25.1
More information about the dev
mailing list