[dpdk-dev] [PATCH] vhost: fix connect hang in client mode

Ilya Maximets i.maximets at samsung.com
Thu Jul 21 13:14:59 CEST 2016


On 21.07.2016 13:37, Ilya Maximets wrote:
> 
> 
> On 21.07.2016 13:13, Yuanhan Liu wrote:
>> On Thu, Jul 21, 2016 at 12:45:32PM +0300, Ilya Maximets wrote:
>>> On 21.07.2016 12:37, Yuanhan Liu wrote:
>>>> On Thu, Jul 21, 2016 at 11:21:15AM +0300, Ilya Maximets wrote:
>>>>> If something abnormal happened to QEMU, 'connect()' can block calling
>>>>> thread (e.g. main thread of OVS) forever or for a really long time.
>>>>> This can break whole application or block the reconnection thread.
>>>>>
>>>>> Example with OVS:
>>>>>
>>>>> 	ovs_rcu(urcu2)|WARN|blocked 512000 ms waiting for main to quiesce
>>>>> 	(gdb) bt
>>>>> 	#0  connect () from /lib64/libpthread.so.0
>>>>> 	#1  vhost_user_create_client (vsocket=0xa816e0)
>>>>> 	#2  rte_vhost_driver_register
>>>>> 	#3  netdev_dpdk_vhost_user_construct
>>>>> 	#4  netdev_open (name=0xa664b0 "vhost1")
>>>>> 	[...]
>>>>> 	#11 main
>>>>>
>>>>> Fix that by setting non-blocking mode for client sockets for connection.
>>>>>
>>>>> Fixes: 64ab701c3d1e ("vhost: add vhost-user client mode")
>>>>
>>>> Thanks for spotting and fixing yet another bug!
>>>>
>>>>>  
>>>>> +static int
>>>>> +vhost_user_connect_nonblock(int fd, struct sockaddr *un, size_t sz)
>>>>
>>>> I don't quite understand why this is needed: connect() with O_NONBLOCK
>>>> flag set is not enough?
>>>
>>> There is a little issue with non-blocking connect() call. Connection
>>> establishing may be started but '-1' returned with 'errno = EINPROGRESS'.
>>> In this case we must wait on fd until it will be available for writing.
>>> After that we need to check current status of connection using getsockopt().
>>>
>>> I don't sure that we're able to get such situation, but it's documented,
>>> and, I think, we should handle it.
>>>
>>> See 'man connect' for details.
>>
>> I see. Thanks.
>>
>> But basically, I don't like the way of introduing yet another
>> fdset here. I'm wondering we could leverage current fdset code
>> to achieve that. This might need some work though.
>>
>> So how about making it simple and stupid at this stage: sleep a
>> while (maybe 1ms, or maybe 1s) when that happens, and give up
>> when the connection is still not established?
> 
> Hmm, how about this fixup:
> ------------------------------------------------------------------------------
> diff --git a/lib/librte_vhost/vhost_user/vhost-net-user.c b/lib/librte_vhost/vhost_user/vhost-net-user.c
> index 8626d13..b0f45e6 100644
> --- a/lib/librte_vhost/vhost_user/vhost-net-user.c
> +++ b/lib/librte_vhost/vhost_user/vhost-net-user.c
> @@ -537,18 +537,7 @@ vhost_user_connect_nonblock(int fd, struct sockaddr *un, size_t sz)
>  	errno = EINVAL;
>  
>  	ret = connect(fd, un, sz);
> -	if (ret == -1 && errno != EINPROGRESS)
> -		return -1;
> -	if (ret == 0)
> -		goto connected;
> -
> -	FD_ZERO(&fdset);
> -	FD_SET(fd, &fdset);
> -
> -	ret = select(fd + 1, NULL, &fdset, NULL, &tv);
> -	if (!ret)
> -		errno = ETIMEDOUT;
> -	if (ret != 1)
> +	if (ret < 0 && errno != EISCONN)
>  		return -1;
>  
>  	ret = getsockopt(fd, SOL_SOCKET, SO_ERROR, &so_error, &len);
> @@ -558,7 +547,6 @@ vhost_user_connect_nonblock(int fd, struct sockaddr *un, size_t sz)
>  		return -1;
>  	}
>  
> -connected:
>  	flags = fcntl(fd, F_GETFL, 0);
>  	if (flags < 0) {
>  		RTE_LOG(ERR, VHOST_CONFIG,
> ------------------------------------------------------------------------------
> ?
> 
> We will not check the EINPROGRESS, but subsequent 'connect()' will return
> EISCONN if connection already established. getsockopt() is kept just in
> case. Subsequent 'connect()' will happen on the next iteration of
> reconnection cycle (1 second sleep).

I've sent v2 with this changes.

Best regards, Ilya Maximets.


More information about the dev mailing list