[dpdk-dev] [RFC 00/21] mlx5: sharing global MR cache between drivers
Michael Baum
michaelba at nvidia.com
Tue Aug 17 15:44:20 CEST 2021
There are 5 classes of mlx5 drivers (net\eth, RegEx, vDPA, compress and
crypto). The various drivers are registered under the common mlx5 driver
and are managed by it.
The common driver probing calls in a loop to the probe function of each
driver registered to it.
Each driver creates for itself all the objects required for
communication with the hardware and a global MR cache that manages
memory mappings.
The management of the caches separately by the different drivers is not
very efficient. In fact, the same memory is mapped multiple times to the
HW when more than 1 class use the device.
This feature will move management to the common driver in two phases.
Phase 1: sharing HW objects between drivers on the same device
The communication with the hardware - for any MR handle - is conducted
by the Protection Domain, so we are motivated to share it between the
drivers. However, to create it, we need to give the context of the
device, so the context must also be shared between the drivers.
At this point, we will share the next trio between the drivers (CTX, PD,
pdn) to create an infrastructure that will allow sharing of dependent
objects, particularly the global MR cache.
The common driver itself will create this trio individually for all
drivers before calling their probe function. Then, as a parameter to the
probe function, it will give them a pointer to the structure containing
the trio.
Phase 2: sharing global MR cache between drivers on the same device
The common driver will add to the structure containing the trio and the
structure that manages the global MR cache and keep a list of such
structures for memory management. In each driver, each queue will manage
its own local MR cache. If the queue does not find its cache, it will
search the global MR cache shared by all. Caching access will be through
the pointer that the driver received as a parameter in probing.
Michael Baum (21):
net/mlx5: fix shared device context creation error flow
net/mlx5: fix PCI probing error flow
common/mlx5: add context device structure
compress/mlx5: use context device structure
crypto/mlx5: use context device structure
regex/mlx5: use context device structure
net/mlx5: improve probe function on Windows
net/mlx5: improve probe function on Linux
net/mlx5: improve spawn function
net/mlx5: use context device structure
net/mlx5: move NUMA node field to context device
common/mlx5: add ROCE disable in context device creation
vdpa/mlx5: use context device structure
mlx5: update device sent to probing
mlx5: share context device structure between drivers
common/mlx5: add HCA attributes to context device structure
regex/mlx5: use HCA attributes from context device
vdpa/mlx5: use HCA attributes from context device
compress/mlx5: use HCA attributes from context device
crypto/mlx5: use HCA attributes from context device
net/mlx5: use HCA attributes from context device
drivers/common/mlx5/linux/mlx5_common_os.c | 268 ++++++++-
drivers/common/mlx5/mlx5_common.c | 273 +++++++++-
drivers/common/mlx5/mlx5_common.h | 35 +-
drivers/common/mlx5/mlx5_common_private.h | 6 -
drivers/common/mlx5/version.map | 2 +
drivers/common/mlx5/windows/mlx5_common_os.c | 207 ++++++-
drivers/compress/mlx5/mlx5_compress.c | 112 +---
drivers/crypto/mlx5/mlx5_crypto.c | 111 +---
drivers/crypto/mlx5/mlx5_crypto.h | 4 +-
drivers/crypto/mlx5/mlx5_crypto_dek.c | 5 +-
drivers/net/mlx5/linux/mlx5_ethdev_os.c | 8 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 9 +-
drivers/net/mlx5/linux/mlx5_os.c | 543 +++++++++----------
drivers/net/mlx5/linux/mlx5_verbs.c | 55 +-
drivers/net/mlx5/mlx5.c | 85 ++-
drivers/net/mlx5/mlx5.h | 17 +-
drivers/net/mlx5/mlx5_devx.c | 35 +-
drivers/net/mlx5/mlx5_flow.c | 6 +-
drivers/net/mlx5/mlx5_flow_aso.c | 24 +-
drivers/net/mlx5/mlx5_flow_dv.c | 51 +-
drivers/net/mlx5/mlx5_flow_verbs.c | 4 +-
drivers/net/mlx5/mlx5_mr.c | 14 +-
drivers/net/mlx5/mlx5_txpp.c | 27 +-
drivers/net/mlx5/windows/mlx5_ethdev_os.c | 14 +-
drivers/net/mlx5/windows/mlx5_os.c | 285 ++--------
drivers/regex/mlx5/mlx5_regex.c | 74 +--
drivers/regex/mlx5/mlx5_regex.h | 23 +-
drivers/regex/mlx5/mlx5_regex_control.c | 12 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 18 +-
drivers/regex/mlx5/mlx5_rxp.c | 64 ++-
drivers/vdpa/mlx5/mlx5_vdpa.c | 210 +------
drivers/vdpa/mlx5/mlx5_vdpa.h | 4 +-
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 19 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 6 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 13 +-
drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 10 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 16 +-
37 files changed, 1414 insertions(+), 1255 deletions(-)
--
2.25.1
More information about the dev
mailing list