[dpdk-dev] [PATCH 00/13] Introducing EAL Bus-Device-Driver Model
Shreyansh Jain
shreyansh.jain at nxp.com
Sun Dec 4 11:11:15 CET 2016
DPDK has been inherently a PCI inclined framework. Because of this, the
design of device tree (or list) within DPDK is also PCI inclined. A
non-PCI device doesn't have a way of being expressed without using hooks
started from EAL to PMD.
:: Overview of the Proposed Changes ::
Assuming the below graph for a computing node:
device A1
|
+==.===='==============.============+ Bus A.
| `--> driver A11 \
device A2 `-> driver A12 \______
|CPU |
/`````
device B1 /
| /
+==.===='==============.============+ Bus B`
| `--> driver B11
device B2 `-> driver B12
- One or more buses are connected to a CPU (or core)
- One or more devices are conneted to a Bus
- Drivers are running instances which manage one or more devices
- Bus is responsible for identifying devices (and interrupt propogation)
- Driver is responsible for initializing the device
In context of DPDK EAL:
- rte_bus, represents a Bus. An implementation of a physical bus would
instantiate this class.
- Buses are registered just like a PMD - RTE_REGISTER_BUS()
`- Thus, implementation for PCI would instantiate a rte_bus, give it a
name and provide scan/match hooks.
- Currently, priority of RTE_REGISTER_BUS constructor has been set to
101 to make sure bus is registered *before* drivers are.
- Each registered bus is part of a doubly list.
-- Each device refers to rte_bus on which it belongs
-- Each driver refers to rte_bus with which it is associated
-- Device and Drivers lists are part of rte_bus
-- NO global device/driver list would exist
- When a PMD wants to register itself, it would 'add' itself to an
existing bus. Which essentially converts to adding the driver to
a bus specific driver_list.
- Bus would perform a scan and 'add' devices scanned to its list.
- Bus would perform a probe and link devices and drivers on each bus and
invoking a series of probes
`-- There are some parallel work for combining scan/probe in EAL [5]
and also for doing away with a independent scan function all
together [6].
The view would be almost like:
__ rte_bus_list
/
+----------'---+
|rte_bus |
| driver_list------> device_list for this bus
| device_list----
| scan() | `-> driver_list for this bus
| match() |
| probe() |
| |
+--|------|----+
_________/ \_________
+--------/----+ +-\---------------+
|rte_device | |rte_driver |
| *rte_bus | | *rte_bus |
| rte_driver | | probe() |
| | | remove() |
| devargs | | |
+---||--------+ +---------|||-----+
|| '''
| \ \\\
| \_____________ \\\
| \ |||
+------|---------+ +----|----------+ |||
|rte_pci_device | |rte_xxx_device | |||
| PCI specific | | xxx device | |||
| info (mem,) | | specific fns | / | \
+----------------+ +---------------+ / | \
_____________________/ / \
/ ___/ \
+-------------'--+ +------------'---+ +--'------------+
|rte_pci_driver | |rte_vdev_driver | |rte_xxx_driver |
| PCI id table, | | <probably, | | .... |
| other driver | | nothing> | +---------------+
| data | | ... |
| probe() | +----------------+
| remove() |
+----------------+
In continuation to the RFC posted on 17/Nov [9],
A series of patches is being posted which attempts to create:
1. A basic bus model
`- define rte_bus and associated methods/helpers
`- test infrastructure to test the Bus infra
2. Changes in EAL to support PCI as a bus
`- a "pci" bus is registered
`- existing scan/match/probe are modified to allow for bus integration
`- PCI Device and Driver list, which were global entities, have been
moved to rte_bus->[device/driver]_list
I have sanity tested this patch over a XeonD X552 available with me, as
well as part of PoC for verifying NXP's DPAA2 PMD (being pushed out in a
separate series). Exhaustive testing is still pending.
:: Brief about Patch Layout ::
0001: Container_of patch from [3]
0002~0003: Introducing the basic Bus model and associated test case
0004: Add scan and match callbacks for the Bus and updated test case
0005: Support insertion of device rather than addition to tail
0006: Integrate bus scan/match with EAL, without any effective driver
0007: rte_pci_driver->probe replaced with rte_driver->probe
0008: Integrate probe of drivers with EAL
0009: Split the existing PCI probe into match and probe
0010: Make PCI probe/match work on rte_driver/device rather than
rte_pci_device/rte_pci_driver
0011: Patch from Ben [8], part of series [2]
0012~0013: Enable Scan/Match/probe on Bus from EAL and remove unused
functions and lists
:: Pending Changes/Caveats ::
0. eth_dev still contains rte_pci_device. I am banking on Jan's patches [1]
for conversion to a macro (ETH_DEV_PCI_DEV) and subsequent replacement
of all pci_dev usage in eth_dev.
1. One of the major changes pending, as against proposed in RFC, is the
removal of eth_driver.
Being a large change, and independent one, this would be done in a
separate series of patches. []
2. app/test/test_pci.c is still not modified. It is rendered uncompilable
as most of the APIs it was calling don't exist anymore. I will push a
v2 which would include a complete re-write of test_pci (or, if someone
can help me that, it would be awesome).
3. This patchset only moves the PCI into a bus. And, that movement is also
currently part of the EAL (lib/librte_eal/linux)
- there was an open question in RFC about where to place the PCI bus
instance - whether in drivers/bus/... or in lib/librte_bus/... or
lib/librte_eal/...; This patch uses the last option. But, movement
only impacts placement of Makefiles. Please convey your reservations
for current placement.
- It also impacts the point (8) about priority use in constructor
4. Though the implementation for bus is common for Linux and BSD, the PCI
bus implementation has been done/tested only for Linux.
5. There was a suggestion from Jan Blunk about a helper iterator within the
rte_bus. I will send this out in v2.
6. Cyptodev and VDEV changes are still pending. Jan has already conveyed
that he would be working on the VDEV part. I will work on the Crypto
part and target that for v2.
7. The overall layout for driver probing has changed a little.
earlier, it was:
rte_eal_init()
`-> rte_eal_pci_probe() (and parallel for VDEV)
`-> rte_pci_driver->probe()
`-> eth_driver->eth_dev_init()
now, it would be:
rte_eal_bus_probe()
`-> bus->prove() (*) <- Iterator for PCI device/driver;
| old rte_eal_pci_probe()
`-> rte_driver->probe() <- create eth_dev
`-> rte_pci_driver->probe() <- PMD init; old eth_dev_init
(*) Buses don't do probe. But in DPDK, EAL was doing this. This patch
pushes that responsibility a layer down, towards rte_bus.
See Patch 0008 for more explanantion
Also, rte_driver->probe() creating eth_dev certainly sounds a little
wrong - but, I would like to get your opinion on how to lay this
order of which layer ethernet device corresponds to.
1) Which layer should allocate eth_dev?
`-> My take: rte_driver->probe()
2) which layer should fill the eth_dev?
`-> My take: rte_xxx_driver->probe()
3) Is init/uninit better name for rte_xxx_driver()->probe() if all
they do is initialize the ethernet device?
8. RTE_REGISTER_BUS has been declared with contructor priority of 101
It is important that Bus is registered *before* drivers are registered.
Only way I could find to assure that was via
__attribute(contructor(priority)) of GCC. I am not sure how it would
behave on other compilers. Any suggestions?
:: Some known issues ::
- Checkpatch is failing on patch 0001
- app/test/test_pci.c compilation is failing, if enabled. By default it
seems to be disabled (CONFIG_RTE_APP_TEST_RESOURCE_TAR)
- Bump to librte_eal version
:: References ::
[1] http://dpdk.org/ml/archives/dev/2016-November/050186.html
[2] http://dpdk.org/ml/archives/dev/2016-November/050622.html
[3] http://dpdk.org/ml/archives/dev/2016-November/050416.html
[4] http://dpdk.org/ml/archives/dev/2016-November/050567.html
[5] http://dpdk.org/ml/archives/dev/2016-November/050628.html
[6] http://dpdk.org/ml/archives/dev/2016-November/050415.html
[7] http://dpdk.org/ml/archives/dev/2016-November/050443.html
[8] http://dpdk.org/ml/archives/dev/2016-November/050624.html
[9] http://dpdk.org/ml/archives/dev/2016-November/050296.html
Ben Walker (1):
pci: Pass rte_pci_addr to functions instead of separate args
Jan Blunck (1):
eal: define container_of macro
Shreyansh Jain (11):
eal/bus: introduce bus abstraction
test: add basic bus infrastructure tests
eal/bus: add scan and match support
eal/bus: add support for inserting a device on a bus
eal: integrate bus scan and probe with EAL
pci: replace probe and remove handlers with rte_driver
eal: enable probe and remove from bus infrastructure
pci: split match and probe function
eal/pci: generalize args of PCI scan/match towards RTE device/driver
eal: enable PCI bus
eal/pci: remove PCI probe and init calls
app/test/Makefile | 2 +-
app/test/test.h | 2 +
app/test/test_bus.c | 688 ++++++++++++++++++++++++
app/test/test_pci.c | 2 +-
drivers/net/bnx2x/bnx2x_ethdev.c | 12 +-
drivers/net/bnxt/bnxt_ethdev.c | 6 +-
drivers/net/cxgbe/cxgbe_ethdev.c | 6 +-
drivers/net/e1000/em_ethdev.c | 6 +-
drivers/net/e1000/igb_ethdev.c | 12 +-
drivers/net/ena/ena_ethdev.c | 6 +-
drivers/net/enic/enic_ethdev.c | 6 +-
drivers/net/fm10k/fm10k_ethdev.c | 6 +-
drivers/net/i40e/i40e_ethdev.c | 6 +-
drivers/net/i40e/i40e_ethdev_vf.c | 6 +-
drivers/net/ixgbe/ixgbe_ethdev.c | 12 +-
drivers/net/mlx4/mlx4.c | 6 +-
drivers/net/mlx5/mlx5.c | 4 +-
drivers/net/nfp/nfp_net.c | 6 +-
drivers/net/qede/qede_ethdev.c | 16 +-
drivers/net/szedata2/rte_eth_szedata2.c | 6 +-
drivers/net/virtio/virtio_ethdev.c | 4 +-
drivers/net/vmxnet3/vmxnet3_ethdev.c | 6 +-
lib/librte_eal/bsdapp/eal/eal.c | 12 +-
lib/librte_eal/bsdapp/eal/eal_pci.c | 53 +-
lib/librte_eal/bsdapp/eal/rte_eal_version.map | 18 +-
lib/librte_eal/common/Makefile | 2 +-
lib/librte_eal/common/eal_common_bus.c | 285 ++++++++++
lib/librte_eal/common/eal_common_pci.c | 335 +++++++-----
lib/librte_eal/common/eal_private.h | 14 +-
lib/librte_eal/common/include/rte_bus.h | 262 +++++++++
lib/librte_eal/common/include/rte_common.h | 20 +
lib/librte_eal/common/include/rte_dev.h | 14 +
lib/librte_eal/common/include/rte_pci.h | 32 +-
lib/librte_eal/linuxapp/eal/Makefile | 1 +
lib/librte_eal/linuxapp/eal/eal.c | 12 +-
lib/librte_eal/linuxapp/eal/eal_pci.c | 82 +--
lib/librte_eal/linuxapp/eal/rte_eal_version.map | 18 +-
lib/librte_ether/rte_ethdev.c | 16 +-
lib/librte_ether/rte_ethdev.h | 6 +-
39 files changed, 1717 insertions(+), 291 deletions(-)
create mode 100644 app/test/test_bus.c
create mode 100644 lib/librte_eal/common/eal_common_bus.c
create mode 100644 lib/librte_eal/common/include/rte_bus.h
--
2.7.4
More information about the dev
mailing list