[dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes
Raslan Darawsheh
rasland at mellanox.com
Sun Jul 19 09:28:00 CEST 2020
Hi,
> -----Original Message-----
> From: Parav Pandit <parav at mellanox.com>
> Sent: Friday, July 17, 2020 4:49 PM
> To: dev at dpdk.org; grive at u256.net; ferruh.yigit at intel.com; Thomas
> Monjalon <thomas at monjalon.net>
> Cc: Raslan Darawsheh <rasland at mellanox.com>; Ori Kam
> <orika at mellanox.com>; Matan Azrad <matan at mellanox.com>;
> joyce.kong at arm.com; Parav Pandit <parav at mellanox.com>
> Subject: [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple
> classes
>
> This series introduces mlx5 bus to support multiple class of devices
> for a single PCI device.
>
> Motivation and example
> ----------------------
> mlx5 PCI device supports multiple class of devices such as net, vdpa
> and regex devices.
>
> Currently only one pmd (either net or vdpa) can bind to this device.
> This design limits use of PCI device only for single device class.
>
> To support multiple classes simultaneously for a mlx5 PCI device,
> a new mlx5 PCI bus is created. This bus allows binding multiple
> class drivers (such as net, vdpa, regex(future)) to bind to the
> mlx5 PCI bus driver.
>
> Change description
> ------------------
> Patch-1 Introduces RTE_BIT() macro
> Patch-2 Introduces new RTE constructor priority for common
> initialization
> Patch-3 Fixes compilation error
> Patch-4 Define mlx5 class as bit fields
> Patch-5 Uses new RTE common priority
> Patch-6 Adds mlx5 PCI bus
> Patch-7 Implements a mlx5 PCI bus driver
> Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
> API instead of rte PCI bus API
> Patch-9 Removed class check code as its already part of the bus now
>
> Design overview
> ---------------
>
> ----------- ------------ -------------
> | mlx5 | | mlx5 | | mlx5 |
> | net pmd | | vdpa pmd | | regex pmd |
> ----------- ------------ -------------
> \ | /
> \ | /
> \ ------------- /
> \______| mlx5 |_____ /
> | pci bus |
> -------------
> |
> -----------
> | mlx5 |
> | pci dev |
> -----------
>
> - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
> ID table of all related mlx5 PCI devices.
> - mlx5 class driver such as net, vdpa, regex PMD defines its
> specific PCI ID table and mlx5 bus driver probes matching
> class drivers.
> - mlx5 pci bus driver is cental place that validates supported
> class combinations.
> - In future as code evolves, more device setup/cleanup and
> resource creation code moves to mlx5 PCI bus driver.
>
> Alternatives considered
> -----------------------
> 1. Instead of creating mlx5 pci bus, a common driver is
> implemented which exposes class registration API.
> However, bus model fits better with existing DPDK design
> similar to ifpga driver.
> Class registration API need to create a new callbacks
> and ID signature; instead it is better to utilize current
> well defined methods.
>
> 2. Enhance pci core to allow multiple driver binding to
> single rte PCI device.
> This approach is not taken, because peer drivers using
> one PCI device won't be aware of other's presence. This
> requires cross-driver syncronization of who initializes
> common resources (such as irq, eq and more).
> This also requires refcounting common objects etc among
> peer drivers.
> Instead of layered approach delivers and allows putting
> common resource sharing, setup code in common bus driver.
> It also eliminates peer blind zone problem as bottom pci
> bus layer provides necessary setup without any reference
> counting.
>
> 3. In future mlx5 prefers to use RDMA MR cache of the mbuf
> used between net and regex pmd so that same mbuf use across
> multiple device can be possible.
>
> Examples:
> --------
> A user who wish to use a specific class(es) provides list
> of classes at command line such as,
> ./testpmd -w <PCI BDF>,class=net:vdpa
> ./testpmd -w <PCI BDF>,class=vdpa
>
> In future,
> ./testpmd -w <PCI BDF>,class=net:regex
>
> Changelog:
> v6->v7:
> - Updated release notes
> v5->v6:
> - Fixed compilation failure in parallel build for shared lib
> v4->v5:
> - Squash the maintainers update path with other patch which adds the
> bus
> - Addressed comments from Morten Brørup
> - Renamed newly added macro to RTE_BIT64
> - Added doxygen comment section for the macro
> v3->v4:
> - Fixed dma_map error unwinding flow to follow same order for unmap
> v2->v3:
> - Added RTE priority for common driver initialization
> - Addressed comments from Thomas and Asaf
> - Fixed compilation error in glue wrapper
> - Moved pci_driver structure instance as first in driver
> - Removed white spaces at the end of line in diagram
> - Address commnts from Matan
> - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
> - Renamed mlx5_valid_class_combo to mlx5_class_combinations
> - Added cross check for class drivers to support only 3 flags for now
> - Added full stop at the end of comment block
> - Using full names in function names
> - Added new line before function name in multiple functions
> - Added example string to parse for multiple classes
> - Dropped mlx5 prefix from static function
> - Removed empty lines
> - Fixed issue to remove multiple classes for a driver
> - Using define for drv_flags at multiple places
> - Deriving drv_flags based on the class drivers
> - Fixed alignment for id_table
> - Perform dma map on best effort basis for all supported drivers
> - Dynamically build pci id table
> - Using PCI to mlx5 device helper routines
> v1->v2:
> - Addressed most comments from Thomas and Gaetan.
> - Symbols starting with prefix rte_bus_pci_mlx5 may be
> confusing as it may appear as it belong to rte_bus_pci module.
> Hence it is kept as rte_bus_mlx5_pci which matches with other
> modules as mlx5_vdpa, mlx5_net.
> - Dropped 2nd patch and replace with new 6th patch.
> - Avoided new file, added macro to rte_bitops.h
> - Inheriting ret_pci_driver instead of rte_driver
> - Added design and description of the mlx5_pci bus
> - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
> - Use anonymous structure for class search and code changes around it
> - Define static for class comination array
> - Use RTE_DIM to find array size
> - Added OOM check for strdup()
> - Renamed copy variable to nstr_orig
> - Returning negagive error code
> - Returning directly if match entry found
> - Use compat condition check
> - Avoided cutting error message string
> - Use uint32_t datatype instead of enum mlx5_class
> - Changed logic to parse device arguments only once during probe()
> - Added check to fail driver probe if multiple classes register with
> DMA ops
> - Renamed function to parse_class_options
> - Migreate API from rte_driver to rte_pci_driver
>
> Parav Pandit (9):
> eal: introduce macros for getting value for bit
> eal: introduce RTE common initialization level
> common/mlx5: fix empty input style in glue wrappers
> common/mlx5: change mlx5 class enum values as bits
> common/mlx5: use common rte priority
> bus/mlx5_pci: add mlx5 PCI bus
> bus/mlx5_pci: register a PCI driver
> bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver
> common/mlx5: remove class checks from individual driver
>
> MAINTAINERS | 5 +
> doc/guides/rel_notes/release_20_08.rst | 5 +
> drivers/bus/Makefile | 4 +
> drivers/bus/meson.build | 2 +-
> drivers/bus/mlx5_pci/Makefile | 41 ++
> drivers/bus/mlx5_pci/meson.build | 19 +
> drivers/bus/mlx5_pci/mlx5_pci_bus.c | 522 ++++++++++++++++++
> drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++
> .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 +
> drivers/common/mlx5/linux/mlx5_glue.h | 4 +-
> drivers/common/mlx5/mlx5_common.c | 39 +-
> drivers/common/mlx5/mlx5_common.h | 7 +-
> .../common/mlx5/rte_common_mlx5_version.map | 2 -
> drivers/net/mlx5/Makefile | 3 +-
> drivers/net/mlx5/linux/mlx5_os.c | 6 -
> drivers/net/mlx5/meson.build | 2 +-
> drivers/net/mlx5/mlx5.c | 26 +-
> drivers/net/mlx5/mlx5.h | 1 -
> drivers/vdpa/mlx5/Makefile | 3 +-
> drivers/vdpa/mlx5/meson.build | 2 +-
> drivers/vdpa/mlx5/mlx5_vdpa.c | 30 +-
> lib/librte_eal/include/rte_bitops.h | 8 +
> lib/librte_eal/include/rte_common.h | 1 +
> mk/rte.app.mk | 1 +
> 24 files changed, 737 insertions(+), 85 deletions(-)
> create mode 100644 drivers/bus/mlx5_pci/Makefile
> create mode 100644 drivers/bus/mlx5_pci/meson.build
> create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
> create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
>
> --
> 2.26.2
Series applied to next-net-mlx,
Kindest regards,
Raslan Darawsheh
More information about the dev
mailing list