[dpdk-dev] [PATCH v2 04/22] ethdev: enable hotplug on multi-process

[Andrew Rybchenko]

On 06/21/2018 05:00 AM, Qi Zhang wrote:
> We are going to introduce the solution to handle different hotplug
> cases in multi-process situation, it include below scenario:
>
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
>
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
>
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced.
>
> This patch covers the implementation of case 1,2,5,6,7,8.
> Case 3,4 will be implemented on separate patch as well as handshake
> mechanism.
>
> Scenario for Case 1, 2:
>
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
>
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
>
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowed to have
> private device so far.
>
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
>
> APIs changes:
>
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
>
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
>
> Signed-off-by: Qi Zhang <qi.z.zhang at intel.com>
> ---
>
> v2:
> - rename rte_ethdev_mp.* to ethdev_mp.*
> - add experimental tag for rte_eth_dev_attach_private and
>    rte_ethdev_detach_private.
> - move do_eth_dev_attach and do_eth_dev_detach to ethdev_private.h
> - move rte_eth_dev_mp_init before rte_eal_mcfg_complete.
> - fix meson.build.
> - improve commit log.
>
>   lib/librte_eal/common/eal_private.h |   8 ++
>   lib/librte_eal/linuxapp/eal/eal.c   |   7 ++
>   lib/librte_ethdev/Makefile          |   1 +
>   lib/librte_ethdev/ethdev_mp.c       | 198 ++++++++++++++++++++++++++++++++++++
>   lib/librte_ethdev/ethdev_mp.h       |  44 ++++++++
>   lib/librte_ethdev/ethdev_private.h  |  39 +++++++
>   lib/librte_ethdev/meson.build       |   1 +
>   lib/librte_ethdev/rte_ethdev.c      | 184 +++++++++++++++++++++++++++++----
>   lib/librte_ethdev/rte_ethdev.h      |  45 ++++++++
>   lib/librte_ethdev/rte_ethdev_core.h |   5 +
>   10 files changed, 515 insertions(+), 17 deletions(-)
>   create mode 100644 lib/librte_ethdev/ethdev_mp.c
>   create mode 100644 lib/librte_ethdev/ethdev_mp.h
>   create mode 100644 lib/librte_ethdev/ethdev_private.h
>
> diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
> index bdadc4d50..92fa59bed 100644
> --- a/lib/librte_eal/common/eal_private.h
> +++ b/lib/librte_eal/common/eal_private.h
> @@ -258,4 +258,12 @@ int rte_mp_channel_init(void);
>    */
>   void dev_callback_process(char *device_name, enum rte_dev_event_type event);
>   
> +/**
> + * Register mp channel callback functions of ethdev layer.
> + *
> + * @return
> + *  0 on success.
> + *  (<0) on failure.
> + */
> +int rte_eth_dev_mp_init(void);

It looks like it makes cross-dependency between EAL and ethdev.
As far as I can see EAL does not have references to rte_eth_dev
functions yet. It looks really suspicious. The function is declared in
EAL, but implemented in ethdev.
Moreover, it is declared once again in ethdev_mp.h.

<...>

> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
> index 33d12b3a2..2cb6de745 100644
> --- a/lib/librte_ethdev/rte_ethdev_core.h
> +++ b/lib/librte_ethdev/rte_ethdev_core.h
> @@ -622,4 +622,9 @@ struct rte_eth_dev_data {
>    */
>   extern struct rte_eth_dev rte_eth_devices[];
>   
> +extern int ethdev_logtype;
> +#define ethdev_log(level, fmt, ...) \
> +	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)

It looks like it clashes with ethdev logging changes submitted by Ferruh.

Andrew.