[dpdk-dev] [RFC 1/2] doc: introduction to prgdev
Chen Jing D(Mark)
jing.d.chen at intel.com
Fri Jan 20 04:21:37 CET 2017
This is the documentation to describe what prgdev is, how to use
prgdev API and accomplish an image download.
Signed-off-by: Chen Jing D(Mark) <jing.d.chen at intel.com>
---
doc/guides/prog_guide/prgdev_lib.rst | 457 ++++++++++++++++++++++++++++++++++
1 files changed, 457 insertions(+), 0 deletions(-)
create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
new file mode 100644
index 0000000..3917c18
--- /dev/null
+++ b/doc/guides/prog_guide/prgdev_lib.rst
@@ -0,0 +1,457 @@
+Overview
+========
+
+More and more devices start to support on-die programming, which include
+NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
+introduce a API for application to download/upload image to/from the HW.
+So, it's necessary to provide a generic API to perform image download, upload,
+validation, etc.
+
+This RFC patch intends to introduce a DPDK generic programming device layer,
+called prgdev below, to provide an abstract, generic APIs for applications to
+program hardware without knowing the details of programmable devices. From
+driver's perspective, they'll try to adapt their functions to the abstract
+APIs defined in prgdev.
+
+The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
+for FPGA devices, or upgrade firmware for programmble NICs.
+
+But those devices that have the capability to do on-die programming may
+expose versatile talents to apps. Add/delete/update entries in flow table,
+update/overwrite the firmware/microcode, add a new algorithm, update profiles
+, etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
+include the APIs to perform device-unique actions in current stage until it
+became popular. On the contratry, it only provides the simple capability to
+download a segment of buffer(image) from host to on-die device, or vice versa.
+The image could be an algorithm, a flow table in the on-die SRAM, a firmware
+image running in the device, a function within a pipeline, etc. prgdev won't
+try to inteprete the content and it's transparent to prgdev. Apps and devices
+can communicate with a language they can understand each other, which is not
+provided by prgdev to decide what needs to be programmed.
+
+When the set of APIs is introduced, a general question is why we need it in
+DPDK community? Why we can't use offline tool to perform same thing? The answer
+is the prgdev provide a generic, online API to applications, in the meanwile,
+offers a capability to switch driver dynamically when downloaded image changed
+personality and a new driver is required. Comparing offline tool, it have online
+programmability (see below examples); Comparing online tool, it provides a
+generic API for many HWs; Comparing generic online tool/API for many products,
+it provides a capability to switch driver dynamically.
+
+There are various means to reach same goal, we'll try to find the best/better
+way to approach. All above advantages will help prgdev to be a 'better choice'.
+
+
+Scope
+-----
+
+The target devices include but not limited to programmable NIC, FPGA, GPU.
+Any devices, having the capability to store/load a piece of info to/from
+the deivce then changed hardware behavior, and applicable to prgdev programming
+model, could be registered as a prgdev device.
+
+The device can be programmed (dynamic) within DPDK or have been prior programmed
+(static) prior to a DPDK application being launched.
+
+Static example: In a static example a program, bitstream or other firmware
+would have been loaded prior to the start of a DPDK application. The prgdev
+would recognize the firmware loaded, and instantiate higher level interfaces
+(example : PMD, L3 forwarding, flow, etc.).
+
+Dynamic example: In a dynamic example, the programmability of the program,
+bitstream or other firmware would be loaded by the DPDK application upon start
+, or during execution. The prgdev would offer the interface to allow the DPDK
+application to program the device.
+
+Dyanamic bind/unbind
+--------------------
+
+Besides the simple API to upload/download image, the prgdev also introduces a
+mechanism internally to switch drivers and register/unregister device on the
+fly. With this mechanism, apps can use the programmable device, unbind other
+personalities on demand, then program it and bind it back with new personalities
+. Apps can follow below examples to simply complete the whole process.
+
+Note that bind/unbind actions are different concept from that a whole device
+attach/detach. For example, rte_eal_dev_detach(), which will try to detach the
+drivers with device and remove the whole device from specific class of devices
+(ethdev, for example). Then, the whole device is invisible until a new 'probe'
+is activated.
+
+During the whole procedure of image upload/download, prgdev handler is always
+valid and apps can use it operate on programmable device. The reason why unbind
+is necessary is it may break the hardware when programming is in progress while
+other functions are active. Using programmble NIC as an example, it's a little
+impossible to receive/forward packets for hardware while updating firmware. In
+this case, apps need to detach ethdev function before firmware upgrading. Again,
+prgdev handler is still valid, it's function-level detach, different from
+device-level detach. After finishing image download, original function needs to
+attach back, either use same or different drivers, depends on personalities
+changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
+may not, see below examples to get more details.
+
+Another reason to provide bind/unbind action is programmble devices, like FPGA,
+are not identified driver by 'vendor ID' and 'device ID', they might not be
+changed in all the ways, even FPGA is fully programmed. So, it depends on
+internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
+drivers, either a register value or signature, depending on FPGA hardware
+design. In this case, EAL or other bus driver doesn't have the capability to
+identify proper driver for FPGA device. With prgdev introduced, while FPGA is
+always a prgdev, FPGA can use prgdev as primary driver, to find proper function
+driver.
+
+The bind/unbind actions are not trying to break the general initialization
+procedure in DPDK. prgdev still follow current EAL initializatio model to probe
+drivers for devices. See below examples for details.
+
+It's not true that bind/unbind is always necessary for programming. If device
+has a capability to perform specific functions and program in parallel, then
+apps needn't do bind/unbind actions. See 'dynamic program' section below.
+
+Authentication
+--------------
+
+Authentication and permissions for programming the device are outside the scope
+of this API. It is dependent upon a different mechanism not to be supported by
+DPDK. Application can utilize a different mechanism to authenticate user and
+or 'program' as validated for loading.
+
+Misc
+----
+
+PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The
+device can be loaded with a personality, and it would be up to this programmable
+device to load the appropriate "personality driver" based upon an identification
+mechanism to work with the loaded personality.
+
+
+prgdev API
+==========
+
+Below is the major APIs exposed to apps.
+
+- rte_prgdev_get_info
+ Acquire general info of the prgdev.
+- rte_prgdev_open
+ Open a prgdev for later programming. If failed, needs to call below 'unbind'
+ function to detach other functions except prgdev first.
+- rte_prgdev_img_download
+ Download a piece of buffer from host to the device after 'open' succeded.
+- rte_prgdev_img_upload
+ Upload a piece of information from the device to host.
+- rte_prgdev_check_status
+ Even download process is successful, it's necessary to check the status of the
+ downloaded image if worked in expected way by this function. Harware or driver
+ may not have the capability to check that status, so it's optional to driver.
+- rte_prgdev_close
+ After upload/download actions, close the prgdev.
+- rte_prgdev_bind
+ After 'close' action, app can call it to attach the 'new' personality with
+ proper drivers, then expose to app with new functions. prgdev driver will be
+ responsible for finding proper drivers to drive the functions. Desired driver
+ will be responsible for exposing the functions to apps.
+- rte_prgdev_unbind
+ When app are using the device, they need to call this function to unbind the
+ misc functions except prgdev with drivers prior to call 'open' function.
+
+Initialization procedure
+========================
+In this section, there 2 examples, FPGA and programmable NIC, used as examples
+to describe the initialization procedure on how programmable devices are probed
+, initialized and registered as different devices. In which, FPGA has 2
+different possible cases, blank and prior programmed FPGAs. Note that prgdev
+depends and follow on EAL initialization process to probe drivers for prgdev.
+
+Blank FPGA
+----------
+Below is the diagram on how Blank FPGA are probed and registered as a prgdev
+device that expose to apps.
+
+When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor
+ID and device ID, then perform HW initialization if both ID match. After then,
+it registered in prgdev abstract layer as a new device with a set of helper
+function ops.
+
+After that, apps will observe single device: prgdev device. Until now, the FPGA
+device is a simply device without any functionalities from application's
+perspective.
+
+In the following action, app will call 'prgdev bind' function with the device
+ID of the FPGA device to bind the function within the FPGA to specific driver.
+FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware
+and find potential function. Since it's a blank without any prior programming, a
+simply SUCCESS return.
+
+ Abstraction FPGA driver Hardware
+ layer
+ + + +
+ | | |
+ +-----eal pci probe-----> |
+ | | | +------------+
+ | | | | FPGA dev |
+ | +--HW initialization----> | |
++------------+ | | | |+----------+|
+| prgdev dev | <--prgdev_ops_register--+ | || AFU id ||
++------------+ | | | |+----------+|
+ | | | +------------+
+ +-----prgdev bind-------> |
+ | | |
+ | +--Read and parse AFU--->
+ | | |
+ | <--No personality found-+
+ | | |
+ <-------SUCCESS---------+ |
+ | | |
+ + + +
+
+prior programmed FPGA
+---------------------
+
+For a prior programmed FPGA, all execution flow is similar to blank FPGA case,
+except the action of reading signature, as below flow table indicates. It will
+read valid functions, crypto and ethdev, as an example, then find proper driver
+to initialize the functions, then register as different ports.
+
+After initialization, app will observe 3 kinds of devices, a prgdev device, a
+ethdev device and a cryptodev device, which all comes from a single PCIE device.
+
+
+ Abstraction FPGA driver Hardware
+ layer
+ + + +
+ | | |
+ +----eal pci probe------> |
+ | | | +-------------+
+ | | | | FPGA dev |
+ | +--HW initialization----> | |
++------------+ | | | +-------------+
+| prgdev dev | <--prgdev_ops_register--+ | || AFU id ||
++------------+ | | | |-------------|
+ | | | +-------------+
+ +-----prgdev bind-------> |
+ | | |
+ | +--Read and parse AFU--->
+ | | |
+ | <--eth and crypto found-+
++------------+ | | |
+| ethdev port| <---ethdev attach-------+ |
++------------+ | | |
+ | | |
++------------+ | | |
+| crypto port| <---crypto attach-------+ |
++------------+ | | |
+ | | |
+ + + +
+
+
+Programmable NIC
+----------------
+
+Comparing FPGA, programmable NIC has a nature of being a ethdev device. The
+initialization procedure is much more simpler than FPGA. Below is a flow table
+to describe this.
+
+The 'eal pci probe' procedure is similar, it registered itself as a prgdev after
+checking that the NIC is programmable. What is different from FPGA is it
+registered itself as a ethdev device in the following action due to the NIC
+nature.
+
+Then, app still follow the common process to call 'prgdev bind', while the NIC
+driver does nothing and just return a SUCCESS.
+
+ Abstraction NIC driver Hardware
+ layer
+ + + +
+ | | |
+ +----eal pci probe------> |
+ | | | +-------------+
+ | +--HW initialization----> |Programmable |
+ | | | | NIC |
++------------+ | +---Capability check----> +-------------+
+| prgdev dev | <--prgdev_ops_register--+ |
++------------+ | | |
+ | | |
++------------+ | | |
+| ethdev port| <---ethdev attach-------+ |
++------------+ | | |
+ | | |
+ +-----prgdev bind-------> |
+ | | |
+ | | |
+ <--------SUCCESS--------+ |
+ | | |
+ + + +
+
+
+Program procedure
+=================
+
+This section will describe how apps finish an image downloading procedure with
+the prgdev API. Similar to previous section, it will use several FPGA and NIC
+examples to show the common and different steps.
+
+Needs to point out, 'rte_prgdev_img_download' function didn't carry the info
+of the image, or where downloading the image to. The interpretion info of the
+image can be hidden into the buffer itself.
+
+Common programming steps
+------------------------
+
+Before reaching details, it needs to introduce the common steps for apps to
+follow in order to download the image to the hardware.
+
+Before programming, apps need to call function 'rte_prgdev_open' to open the
+programming interface. If failed, either functions residing in PCI-E device is
+in use or the PCI-E device is not ready, the programming steps are different.
+
+Below is the steps that 'open' succeeds:
+1. rte_prgdev_open --> SUCCESS
+2. rte_prgdev_img_download
+3. rte_prgdev_check_status
+4. rte_prgdev_close
+5. rte_prgdev_bind
+
+If failed to open the programming interface, apps need to follow below steps:
+1. rte_prgdev_open --> FAILURE
+2. Close all related functions
+3. ret_prgdev_unbind
+4. rte_prgdev_open --> SUCCESS
+5. rte_prgdev_img_download
+6. rte_prgdev_check_status
+7. rte_prgdev_close
+8. rte_prgdev_bind
+
+Detach functions
+----------------
+
+As discussed before, apps need to follow steps to detach all the other functions
+except prgdev when meeting error for 'open' operation. Below is a FPGA example
+with 'ethdev' and 'cryptodev' function attached.
+
+After meeting error, app needs to close ethdev port and cryptodev port first.
+Then, use 'prgdev unbind' to detach all the functions that registered previously
+. FPGA driver will call each 'detach' function for each registered device class
+respectively. After then, the 'ethdev' and 'cryptodev' ports will not be
+available.
+
+
+ Abstraction FPGA driver Hardware
+ layer
+ + + +
++------------+ | | |
+| prgdev dev | +-------prgdev open-----> |
+| | <-------ERR_IN_USE------+ | +-------------+
++------------+ | | | | FPGA dev |
+ | | | | +---------+ |
++------------+ | | | | | AFU id | |
+| ethdev port| +---rte_ethdev_close----> | | +---------+ |
+| | +------ethdev close-----> +-------------+
++------------+ <---------SUCCESS-------+ |
+ | | |
++------------+ | | |
+| crypto port| +---crypto port close---> |
+| | | +---crypto func close--->
++------------+ <---------SUCCESS-------+ |
+ | | |
+ +-----prgdev unbind-----> |
+ | | |
++ - - - - - -+ | | |
+ ethdev port <-----ethdev detach-----+ |
++- - - - - - + | | |
+ | | |
++ - - - - - -+ | | |
+ crypto port <----cryptodev detach---+ |
++- - - - - - + | | |
+ + + +
+
+Device programming
+------------------
+
+After 'open' prgdev sucessfully, apps follow 'image download', 'check status',
+'close' sequence respectively. After then, they need to 'bind' new functions
+back with 'prgdev bind' call.
+
+Still re-use previous FPGA example, below is the execution sequence. After image
+downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are
+registered and 2 new ports are presented to apps.
+
+
+ Abstraction FPGA driver Hardware
+ layer
+ + + + +-------------+
++------------+ | | | | FPGA dev |
+| | +-----prgdev open-------> | | +---------+ |
+| | <-------SUCCESS---------+ | | | ethdev | |
+| | | | | | +---------+ |
+| | +--prgdev_img_download--> | | +---------+ |
+| | | +----Download image-----> | | crypto | |
+| | | <--------SUCCESS--------+ | +---------+ |
+| | <---------SUCCESS-------+ | +-------------+
+| prgdev dev | | | | |
+| | +---prgdev_chk_status---> | |
+| | | +--Verify image status--> |
+| | | <--------SUCCESS--------+ |
+| | <--------SUCCESS--------+ | |
+| | +-------prgdev close----> | v
+| | | | | +-------------+
+| | +-------prgdev bind-----> | | FPGA dev |
++------------+ | | | | +---------+ |
+ | <--Virtio and flow dev--+ | | virtio | |
++------------+ | | | | +---------+ |
+| ethdev port| <-----ethdev attach-----+ | | +---------+ |
++------------+ | | | | |Flow dev | |
+ | | | | +---------+ |
++------------+ | | | +-------------+
+| Flow port | <-----Flow dev attach---+ |
++------------+ | | |
+ + + +
+
+
+Dyanamic program
+----------------
+
+If prgdev device has a capability to be programmed while functions are in use,
+then we say it supports dynamic program. It's possible to update a profile while
+device is used as a ethdev device.
+
+The device can claim this capability by set a DYNAMIC_PROGRAM flag in function
+'rte_prgdev_get_info'.
+
+The dynamic programming procedure is the same as previous section shows, while
+ignore the steps to 'close' and 'unbind' actions when device is in use.
+
+Note that prgdev driver needs to consider if it can support dynamic program to
+avoid any hardware or software error.
+
+
+Source code layout
+==================
+
+Below is the suggested source code layout.
+- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This
+ doesn't change programmable NIC's directory and remains the same. The PMD
+ driver should include 2 parts, one part is the code to support rte_ethdev API,
+ another part is the code to support rte_prgdev API.
+- 'librte_prgdev' under 'lib' is the new directory store definition and
+ implementation of the generic prgdev API.
+- 'prg' under 'drivers' is the directory to store specfic implementation
+ driver to support rte_prgdev. Each sub-directory could be a prgdev driver to
+ support specific hardware product.
+
+ +---------+ +---------+
+ | drivers | | lib |
+ +---------+ +----+----+
+ | |
+ +----------+-----------+ |
+ | | |
+ +---v---+ +---v----+ +-------v-------+
+ | net | | prg | | librte_prgdev |
+ +---+---+ +---+----+ +---------------+
+ | |
+ +-----+----+ +-----+----+
+ | | | |
++---v----+ +---v----+ +---v----+ +---v----+
+| i40e | | prg NIC| |FPGA ABC| |GPU XYZ |
++--------+ +--------+ +--------+ +--------+
+
+
--
1.7.7.6
More information about the dev
mailing list