[dpdk-dev] [PATCH 2/3] net/vmxnet3: fix vmxnet3 dev_uninit() hot-unplug

[Louis Luo]

Hi Luca,

Thanks for pointing to the document! This "basic requirements" seems to lay the burden on application developers to correctly follow the hot-plug framework's rules, but there seems no mechanism to enforce this procedure (correct me if I'm wrong). What if a buggy application doesn't call stop/close before detaching? 

In addition, in your commit description, you said " The vmxnet3 driver can't call back into dev_close(), and possibly dev_stop(), in dev_uninit()." But actually in vmxnet3_dev_close(), we set hw->adapter_stopped = 1, and in eth_vmxnet3_dev_uninit() we call into vmxnet3_dev_close() ONLY when hw->adapter_stopped == 0. So if the application does meet the hot-plug framework requirement and calls dev_close before calling uninit, eth_vmxnet3_dev_uninit() should not call into vmxnet3_dev_close() again, right? If so, why bother removing this check? 

Or let me ask this way. If a buggy application DOES NOT call dev_close before calling eth_vmxnet3_dev_uninit(), would calling vmxnet3_dev_close() inside eth_vmxnet3_dev_uninit() cause any trouble? And if an application DOES call dev_close before calling eth_vmxnet3_dev_uninit(), have you ever seen vmxnet3_dev_close() being called again and trigger crashes like double-free or something else? If yes, then we need to investigate.

Thanks
Louis

On 9/18/18, 6:15 AM, "Luca Boccassi" <bluca at debian.org> wrote:

    Hi,
    
    The application must already stop and close before detaching (which
    will call uninit). Quoting from the documentation:
    
    "*  Before detaching, they must be stopped and closed.
    
        DPDK applications must call "rte_eth_dev_stop()" and
        "rte_eth_dev_close()" APIs before detaching ports. These functions will
        start finalization sequence of the PMDs."
    
    https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fprog_guide%2Fport_hotplug_framework.html&data=02%7C01%7Cllouis%40vmware.com%7C3d87a5482cd046679d7608d61d68bf9a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C1%7C0%7C636728733078700911&sdata=qBx7zyOJhTmVH0c2ZEBohmIHL6PkYwB6XPaw2epgS0E%3D&reserved=0
    
    On Mon, 2018-09-17 at 19:06 +0000, Louis Luo wrote:
    > Hi Luca,
    > 
    > When eth_vmxnet3_dev_uninit() is called, is it guaranteed that
    > vmxnet3_dev_close/ vmxnet3_dev_stop must have been called? I'm not
    > familiar with the hot-plug procedure, so just wonder if there is any
    > chance that eth_vmxnet3_dev_uninit() is called without calling
    > vmxnet3_dev_close/ vmxnet3_dev_stop.
    > 
    > Thanks,
    > Louis
    > 
    > On 8/16/18, 6:51 AM, "dev on behalf of Luca Boccassi" <dev-bounces at d
    > pdk.org on behalf of bluca at debian.org> wrote:
    > 
    >     The vmxnet3 driver can't call back into dev_close(), and possibly
    >     dev_stop(), in dev_uninit().  When dev_uninit() is called,
    > anything
    >     that those routines would want to clean up has already been
    > released.
    >     Further, for complete cleanup, it is necessary to release any of
    > the
    >     queue resources during dev_close().
    >     This allows a vmxnet3 device to be hot-unplugged without leaking
    >     queues.
    >     
    >     Fixes: dfaff37fc46d ("vmxnet3: import new vmxnet3 poll mode
    > driver implementation")
    >     Cc: stable at dpdk.org
    >     
    >     Signed-off-by: Brian Russell <brussell at brocade.com>
    >     Signed-off-by: Luca Boccassi <bluca at debian.org>
    >     ---
    >      drivers/net/vmxnet3/vmxnet3_ethdev.c | 36 ++++++++++++++++++++
    > --------
    >      1 file changed, 26 insertions(+), 10 deletions(-)
    >     
    >     diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c
    > b/drivers/net/vmxnet3/vmxnet3_ethdev.c
    >     index 2613cd1358..b5d4be5e24 100644
    >     --- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
    >     +++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
    >     @@ -348,16 +348,11 @@ eth_vmxnet3_dev_init(struct rte_eth_dev
    > *eth_dev)
    >      static int
    >      eth_vmxnet3_dev_uninit(struct rte_eth_dev *eth_dev)
    >      {
    >     -	struct vmxnet3_hw *hw = eth_dev->data->dev_private;
    >     -
    >      	PMD_INIT_FUNC_TRACE();
    >      
    >      	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
    >      		return 0;
    >      
    >     -	if (hw->adapter_stopped == 0)
    >     -		vmxnet3_dev_close(eth_dev);
    >     -
    >      	eth_dev->dev_ops = NULL;
    >      	eth_dev->rx_pkt_burst = NULL;
    >      	eth_dev->tx_pkt_burst = NULL;
    >     @@ -803,7 +798,7 @@ vmxnet3_dev_stop(struct rte_eth_dev *dev)
    >      	PMD_INIT_FUNC_TRACE();
    >      
    >      	if (hw->adapter_stopped == 1) {
    >     -		PMD_INIT_LOG(DEBUG, "Device already closed.");
    >     +		PMD_INIT_LOG(DEBUG, "Device already stopped.");
    >      		return;
    >      	}
    >      
    >     @@ -827,7 +822,6 @@ vmxnet3_dev_stop(struct rte_eth_dev *dev)
    >      	/* reset the device */
    >      	VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
    > VMXNET3_CMD_RESET_DEV);
    >      	PMD_INIT_LOG(DEBUG, "Device reset.");
    >     -	hw->adapter_stopped = 0;
    >      
    >      	vmxnet3_dev_clear_queues(dev);
    >      
    >     @@ -837,6 +831,30 @@ vmxnet3_dev_stop(struct rte_eth_dev *dev)
    >      	link.link_speed = ETH_SPEED_NUM_10G;
    >      	link.link_autoneg = ETH_LINK_FIXED;
    >      	rte_eth_linkstatus_set(dev, &link);
    >     +
    >     +	hw->adapter_stopped = 1;
    >     +}
    >     +
    >     +static void
    >     +vmxnet3_free_queues(struct rte_eth_dev *dev)
    >     +{
    >     +	int i;
    >     +
    >     +	PMD_INIT_FUNC_TRACE();
    >     +
    >     +	for (i = 0; i < dev->data->nb_rx_queues; i++) {
    >     +		void *rxq = dev->data->rx_queues[i];
    >     +
    >     +		vmxnet3_dev_rx_queue_release(rxq);
    >     +	}
    >     +	dev->data->nb_rx_queues = 0;
    >     +
    >     +	for (i = 0; i < dev->data->nb_tx_queues; i++) {
    >     +		void *txq = dev->data->tx_queues[i];
    >     +
    >     +		vmxnet3_dev_tx_queue_release(txq);
    >     +	}
    >     +	dev->data->nb_tx_queues = 0;
    >      }
    >      
    >      /*
    >     @@ -845,12 +863,10 @@ vmxnet3_dev_stop(struct rte_eth_dev *dev)
    >      static void
    >      vmxnet3_dev_close(struct rte_eth_dev *dev)
    >      {
    >     -	struct vmxnet3_hw *hw = dev->data->dev_private;
    >     -
    >      	PMD_INIT_FUNC_TRACE();
    >      
    >      	vmxnet3_dev_stop(dev);
    >     -	hw->adapter_stopped = 1;
    >     +	vmxnet3_free_queues(dev);
    >      }
    >      
    >      static void
    >     -- 
    >     2.18.0
    >     
    >     
    > 
    
    -- 
    Kind regards,
    Luca Boccassi