[dpdk-dev] [PATCH v2 04/22] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> We are going to introduce the solution to handle different hotplug
> cases in multi-process situation, it include below scenario:
> 
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
> 
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
> 
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced.
> 
> This patch covers the implementation of case 1,2,5,6,7,8.
> Case 3,4 will be implemented on separate patch as well as handshake
> mechanism.
> 
> Scenario for Case 1, 2:
> 
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
> 
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
> 
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowed to have
> private device so far.
> 
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
> 
> APIs changes:
> 
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
> 
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang at intel.com>
> ---

<snip>

> +	memset(&da, 0, sizeof(da));
> +
> +	if (rte_devargs_parse(&da, "%s", devargs)) {
> +		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
> +		return -EINVAL;
> +	}
> +
> +	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
> +	if (ret) {
> +		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
> +			   da.bus->name, da.name);
> +		free(da.args);
> +		return ret;
> +	}
> +
> +	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
> +		ethdev_log(ERR, "failed to attach to port %d, this is a pmd issue\n",
> +			   port_id);
> +		return -ENODEV;

^^^ Leaking da.args here?

> +	}
> +	free(da.args);
> +	return 0;
> +}
> +
> +static int handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
> +{
> +	RTE_SET_USED(msg);
> +	RTE_SET_USED(peer);
> +	return -ENOTSUP;
> +}
> +
> +static int handle_primary_response(const struct rte_mp_msg *msg, const void *peer)
> +{

<snip>

> +	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> +	if (ret) {
> +		ethdev_log(ERR, "rte_mp_request_sync failed\n");
> +		return ret;
> +	}
> +
> +	req->result = 0;
> +	for (i = 0; i < mp_reply.nb_received; i++) {
> +		struct eth_dev_mp_req *resp =
> +			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
> +		if (resp->result) {
> +			req->result = resp->result;
> +			break;
> +		}
> +	}

Do we care if nb_sent != nb_received?

> +
> +	return 0;
> +}
> +
> +int rte_eth_dev_mp_init(void)
> +{
> +
> +	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> +		if (rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,

<snip>

> +/**
> + * this is a synchronous wrapper for secondary process send
> + * request to primary process, this is invoked when an attach
> + * or detach request issued from primary.
> + */
> +int rte_eth_dev_request_to_primary(struct eth_dev_mp_req *req);
> +
> +/**
> + * this is a synchronous wrapper for primary process send
> + * request to secondary process, this is invoked when an attach
> + * or detach request issued from secondary process.
> + */
> +int rte_eth_dev_request_to_secondary(struct eth_dev_mp_req *req);

Nitpicking,  but the two above functions aren't used outside ethdev 
library. You can probably drop the rte_ prefix.

> +
> +/* Register mp channel callback functions of ethdev layer.*/
> +int rte_eth_dev_mp_init(void);

I don't quite understand what you're doing here. (Or rather, i 
understand the intention, but i don't understand the implementation :) )

This function is meant to be called from EAL at startup. First of all, 
why is it declared twice (once in eal_private, once in ethdev_private)?

Second of all, ethdev is a library, but this function is called from 
EAL. Which means it cannot be in a private header (nor should it be 
declared in EAL), and you cannot even call it from EAL because that 
would introduce a circular dependency between EAL and ethdev.

So, this needs to be redone the other way around - have ethdev register 
itself with EAL, and get called at some point, in a generic way (e.g. 
see how bus probe works for example). I don't know what this would look 
like - maybe some kind of generic multiprocess init?

-- 
Thanks,
Anatoly