[dpdk-dev] [PATCH 00/11] eal: allow virtual pmd drivers as shared lib

[Olivier Matz]

The main goal of the following commits is to be able to load a PMD
virtual device driver as a shared library (DPDK extension). Today it is
already possible to load an external PMD PCI driver (ex: memnic,
virtio-net-pmd, ...), but the DPDK framework does not allow to load a
virtual device driver in this way. For instance, adding the support of
host-side in memnic would require these patches.

How it worked before the patch
------------------------------

Example of what occurs when we started testpmd:

  testpmd -c 0x15 -n 3 \
    --proc-type=primary --huge-dir=/mnt/huge --use-device=eth_ring0 \
    --use-device=eth_ring1 \
    --use-device=02:00.0 \
    -- -i --port-topology=chained

For each "--use-device" option, we call eal_dev_whitelist_add_entry():

  main()
    rte_eal_init()
      eal_parse_args()
        eal_dev_whitelist_add_entry()

Each device is appended in a global string. After parsing all the
"--use-device" argumentss, we have dev_list_str="eth_ring0,eth_ring1,".

Then, after all eal arguments are parsed, eal_dev_whitelist_parse() is
called:

  main()
    rte_eal_init()
      eal_parse_args()
        eal_dev_whitelist_parse()
          is_valid_wl_entry()

It removes the extra ',' at the end of dev_list_str. Then, it splits
each device name and its argument (separated by ';'). Each device is
checked by is_valid_wl_entry(). It checks that the PCI identifier is
correct or that the name of the device starts with a known prefix
("eth_ring", "eth_pcap" or "eth_xenvirt", defined statically in eal code).

Then, rte_eal_pci_init() is called, it scans the PCI bus:

  main()
    rte_eal_init()
      rte_eal_pci_init()

After that, rte_eal_non_pci_ethdev_init() tries all combination:
"eth_ring0", "eth_ring1", ..., "eth_ring31", "eth_pcap0", ...,
"eth_pcap31", ..., "-nodev-0", ..., "-nodev-31". For each, test if it
is in whitelist or not:

  main()
    rte_eal_init()
      rte_eal_non_pci_ethdev_init()
        eal_dev_is_whitelisted(name, &params) # fills params
        dev_types[i].init_fn(name, params) [ex: rte_pmd_ring_init()]
          ...
            rte_eth_dev_allocate()

To check that, eal_dev_is_whitelisted() browse the list of devices in
the whitelist. Then, the init function of the device allocates the
ethernet device structure (attributing a port_id).

The main application then calls rte_pmd_init_all(). It initializes all
poll-mode drivers. Each driver calls rte_eth_driver_register(), like in
this example:

  main()
    rte_pmd_init_all()
      rte_igb_pmd_init()
        rte_eth_driver_register()

The probing of the PCI bus is done using rte_eal_pci_probe(), which
calls pci_probe_all_drivers(dev) for each PCI device:

  main()
    rte_eal_pci_probe()
      pcidev_is_whitelisted(dev)
      pci_probe_all_drivers(dev)
        rte_eal_pci_probe_one_driver(dev, driver)
          driver->devinit(driver, dev) [rte_eth_dev_init()]
            rte_eth_dev_allocate()
            eth_drv->eth_dev_init(eth_drv, eth_dev) [ex: eth_em_dev_init()]

List of problems
----------------

- pmd_ring, pmd_pcap and pmd_xenvirt are referenced in eal code:
  - it's not possible to add a new virtual pmd dynamically
  - eal compilation depends on these pmd... but the pmd depends on eal.
  - it's referenced twice with duplicated values: eal_common_nonpci_devs.c
    and eal_common_whitelist.c

- the parsing of virtual devices arguments is complex:
  - they are all appended in a global string then split again

- there are 2 ways to do the same thing, like in the following example:
  - --use-device="eth_ring0,eth_pcap0;iface=ixgbe0"
  - --use-device="eth_ring0" --use-device="eth_pcap0;iface=ixgbe0"

- the same --use-device option is used for both pci whitelist and
  virtual devices

- files and functions related to virtual devices are called 'non_pci',
  'vdev' would be clearer (all non_pci devices aren't virtual devices).

- using ";" to separate a device and its argument is not a good idea in
  a command line argument as it can be used in shell to separate
  commands.

- it is not possible to use the blacklist mode (bind all devices) while
  we add virtual devices.

Summary of the changes introduced by the patchset
-------------------------------------------------

- allow to register a virtual device driver from a dpdk extension
  provided as a shared library

- embed all library symbols in dpdk library.

- remove references to rte_pmd_ring, rte_pmd_pcap and rte_pmd_xenvirt in
  eal code

- add a new rte_devargs file in eal that unifies the code storing the
  user arguments pci-blacklist, pci-whitelist, and virtual devices
  arguments in one file

- rework eal user arguments
  - "--use-device" becomes "--pci-whitelist" and "--vdev"
  - replace ";" by "," when parsing device args

- support start-up arguments for PCI devices


Olivier Matz (11):
  mk: use whole-archive option when creating dpdk binaries
  devices-args: introduce rte_devargs in eal
  devices-args: use rte_devargs and remove old whitelist code
  devices-args: add a dump_devargs command in basic test application
  pci: rename device_list as pci_device_list
  vdev: rename eal_common_nonpci_devs.c as eal_common_vdev.c
  vdev: allow external registration of virtual device drivers
  device-args: use a comma instead of semicolon to separate key/values
  device-args: replace use-device eal option by pci-whitelist and vdev
  device-args: allow to provide per pci device command line arguments
  testpmd: add several dump commands, useful for debug

 app/test-pmd/cmdline.c                         | 114 +++++++++++++
 app/test/Makefile                              |   1 +
 app/test/commands.c                            |   9 +-
 app/test/test.h                                |   1 +
 app/test/test_devargs.c                        | 132 ++++++++++++++
 app/test/test_eal_flags.c                      |  70 ++------
 app/test/test_kvargs.c                         |  14 +-
 app/test/test_pci.c                            |  55 +++---
 app/test/test_pmd_ring.c                       |   6 +-
 lib/librte_eal/common/Makefile                 |   2 +-
 lib/librte_eal/common/eal_common_devargs.c     | 153 +++++++++++++++++
 lib/librte_eal/common/eal_common_nonpci_devs.c |  93 ----------
 lib/librte_eal/common/eal_common_pci.c         |  98 +++++------
 lib/librte_eal/common/eal_common_vdev.c        | 107 ++++++++++++
 lib/librte_eal/common/eal_common_whitelist.c   | 227 -------------------------
 lib/librte_eal/common/include/eal_private.h    |  42 +----
 lib/librte_eal/common/include/rte_devargs.h    | 140 +++++++++++++++
 lib/librte_eal/common/include/rte_pci.h        |  20 +--
 lib/librte_eal/common/include/rte_vdev.h       |  90 ++++++++++
 lib/librte_eal/linuxapp/eal/Makefile           |   4 +-
 lib/librte_eal/linuxapp/eal/eal.c              | 114 +++++++------
 lib/librte_eal/linuxapp/eal/eal_ivshmem.c      |   2 +-
 lib/librte_eal/linuxapp/eal/eal_pci.c          |  16 +-
 lib/librte_kvargs/rte_kvargs.h                 |   6 +-
 lib/librte_pmd_pcap/rte_eth_pcap.c             |  16 +-
 lib/librte_pmd_pcap/rte_eth_pcap.h             |   8 -
 lib/librte_pmd_ring/rte_eth_ring.c             |  17 +-
 lib/librte_pmd_ring/rte_eth_ring.h             |   6 +-
 lib/librte_pmd_xenvirt/rte_eth_xenvirt.c       |  14 +-
 lib/librte_pmd_xenvirt/rte_eth_xenvirt.h       |   4 +-
 mk/rte.app.mk                                  |   5 +
 31 files changed, 987 insertions(+), 599 deletions(-)
 create mode 100644 app/test/test_devargs.c
 create mode 100644 lib/librte_eal/common/eal_common_devargs.c
 delete mode 100644 lib/librte_eal/common/eal_common_nonpci_devs.c
 create mode 100644 lib/librte_eal/common/eal_common_vdev.c
 delete mode 100644 lib/librte_eal/common/eal_common_whitelist.c
 create mode 100644 lib/librte_eal/common/include/rte_devargs.h
 create mode 100644 lib/librte_eal/common/include/rte_vdev.h

-- 
1.8.5.3