[dpdk-dev] [PATCH 6/6] doc: introduction to prgdev

[Chen, Jing D]

Hi, Thomas,

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon at 6wind.com]
> Sent: Monday, March 6, 2017 11:27 PM
> To: Chen, Jing D <jing.d.chen at intel.com>
> Cc: dev at dpdk.org; Liang, Cunming <cunming.liang at intel.com>; Rogers,
> Gerald <gerald.rogers at intel.com>; Wiles, Keith <keith.wiles at intel.com>;
> Richardson, Bruce <bruce.richardson at intel.com>; Mcnamara, John
> <john.mcnamara at intel.com>
> Subject: Re: [dpdk-dev] [PATCH 6/6] doc: introduction to prgdev
> 
> 2017-03-02 12:03, Chen Jing D:
> > +Overview
> > +========
> 
> I think the first review pass of this series must be focused on the overview
> you give here (thanks for the detailed explanations).
> 
> I'll try to sum up while commenting.
> 
> [...]
> The target is programmable devices.
> 
> > +The major purpose of prgdev is to help DPDK users to load/upgrade RTL
> > +images for FPGA devices, or upgrade firmware for programmble NICs.
> 
> This is a control plane feature.
> You want to have it in DPDK to allow dynamic programmation while running
> the device, right?

Exactly right.

> 
> [...]
> > +When the set of APIs is introduced, a general question is why we need
> > +it in DPDK community?
> 
> Good question :)
> 
> [...]
> > +Any devices, having the capability to store/load a piece of info
> > +to/from the deivce then changed hardware behavior, and applicable to
> > +prgdev programming model, could be registered as a prgdev device.
> > +
> > +The device can be programmed (dynamic) within DPDK or have been prior
> > +programmed
> > +(static) prior to a DPDK application being launched.
> [...]
> > +Besides the simple API to upload/download image, the prgdev also
> > +introduces a mechanism internally to switch drivers and
> > +register/unregister device on the fly. With this mechanism, apps can
> > +use the programmable device, unbind other personalities on demand,
> then program it and bind it back with new personalities.
> > +Apps can follow below examples to simply complete the whole process.
> 
> I disagree about the specific bind/unbind for prgdev.
> We must improve binding inside DPDK for every devices.

First of all, let us try to imagine what is the safe status for device before apps
intending to download some image to it - apps wouldn't operate on the device, 
any behaviors like configuring registers, receive/transmit data may impair the
device or make the download failed. 
Following first answer to prevent app accessing device during image
downloading, how can we achieve that? Detach drivers with device is a smart
idea, right? But the problem is how can apps use prgdev API to download image
after all drivers detached with the device?
So, the final question is how can we just detached others driver except prgdev
one? I can't find answer in current DPDK framework, that's why I'd like to introduce
bind/unbind functions to detach other drivers from the device.

I'm open to this problem. If any suggestion or mechanism can help on it, I can
remove these 2.

BTW, not all devices or image download actions need the device to detach the 
device, depending on hardware implementation.

> 
> > +Note that bind/unbind actions are different concept from that a whole
> > +device attach/detach. For example, ``rte_eal_dev_detach()``, which
> > +will try to detach the drivers with device and remove the whole
> > +device from specific class of devices (ethdev, for example). Then, the
> whole device is invisible until a new 'probe'
> > +is activated.
> 
> I do not understand.

See above explanations.

> 
> > +During the whole procedure of image upload/download, prgdev handler
> > +is always valid and apps can use it operate on programmable device.
> > +The reason why unbind is necessary is it may break the hardware when
> > +programming is in progress while other functions are active. Using
> > +programmble NIC as an example, it's a little impossible to
> > +receive/forward packets for hardware while updating firmware. In this
> > +case, apps need to detach ethdev function before firmware upgrading.
> > +Again, prgdev handler is still valid, it's function-level detach,
> > +different from device-level detach. After finishing image download,
> > +original function needs to attach back, either use same or different
> > +drivers, depends on personalities changed or not. For a programmble NIC,
> the driver may be the same. For FPGA, it may not, see below examples to get
> more details.
> 
> If the personality of the device is changed, it must be seen as a new device
> from e.g. the ethdev point of view, while keeping the same prgdev device.
> In other words, a device can have several interfaces at the same time (ethdev,
> cryptodev, eventdev, prgdev, whatever).
> I think we can dynamically create/destroy some interfaces while keeping
> track of the underlying device.

Fully agree. But current PCI device with 'vendor_id' and 'device_id' ony can
be bind to single driver at a time. So I add 'rte_prgdev_allocate/release' to
support register/unregister prgdev dynamically without BDF.

> 
> > +Another reason to provide bind/unbind action is programmble devices,
> > +like FPGA, are not identified driver by 'vendor ID' and 'device ID',
> > +they might not be changed in all the ways, even FPGA is fully
> > +programmed. So, it depends on internal mechanism of FPGA, not 'vendor
> > +ID' and 'device ID' to identify proper drivers, either a register
> > +value or signature, depending on FPGA hardware design. In this case,
> > +EAL or other bus driver doesn't have the capability to identify
> > +proper driver for FPGA device. With prgdev introduced, while FPGA is
> > +always a prgdev, FPGA can use prgdev as primary driver, to find proper
> function driver.
> 
> You mean prgdev should help the bus layer to map the right driver interface?
> It looks weird and dangerous. The standard way to identify a PCI device is to
> look at its IDs. Other unknown methods must be strongly discussed.

For programmable Ethernet device, it's not truce. But for FPGA, it's. When FPGA
is produced, the device ID indicate what model it is and won't be changed
anyway, even being reprogrammed. It used some not-generic mechanism, like
AFU id to distinguish the personalities. So, for FPGA, a prgdev driver can be used
as primary driver to identify personalities and then register to specific devices.