[dpdk-dev] pthread_barrier_deadlock in -rc1

Burakov, Anatoly anatoly.burakov at intel.com
Wed May 2 11:30:56 CEST 2018


On 02-May-18 9:50 AM, Tan, Jianfeng wrote:
> Hi Maxime,
> 
> 
> On 5/2/2018 4:34 PM, Maxime Coquelin wrote:
>> Hi Jiangfeng,
>>
>> On 05/02/2018 10:19 AM, Tan, Jianfeng wrote:
>>>
>>>
>>> On 5/1/2018 2:46 AM, Olivier Matz wrote:
>>>> Hi Maxime,
>>>>
>>>> Le 30 avril 2018 17:45:52 GMT+02:00, Maxime Coquelin 
>>>> <maxime.coquelin at redhat.com> a écrit :
>>>>> Hi Olivier,
>>>>>
>>>>> On 04/24/2018 04:46 PM, Olivier Matz wrote:
>>>>>> Some parts of dpdk use their own management threads. Most of the
>>>>> time,
>>>>>> the affinity of the thread is not properly set: it should not be
>>>>> scheduled
>>>>>> on the dataplane cores, because interrupting them can cause packet
>>>>> losses.
>>>>>> This patchset introduces a new wrapper for thread creation that does
>>>>>> the job automatically, avoiding code duplication.
>>>>>>
>>>>>> v3:
>>>>>> * new patch: use this API in examples when relevant.
>>>>>> * replace pthread_kill by pthread_cancel. Note that pthread_join()
>>>>>>     is still needed.
>>>>>> * rebase: vfio and pdump do not have control pthreads anymore, and
>>>>> eal
>>>>>>     has 2 new pthreads
>>>>>> * remove all calls to snprintf/strlcpy that truncate the thread name:
>>>>>>     all strings lengths are already < 16.
>>>>>>
>>>>>> v2:
>>>>>> * set affinity to master core if no core is off, as suggested by
>>>>>>     Anatoly
>>>>>>
>>>>>> Olivier Matz (5):
>>>>>>     eal: use sizeof to avoid a double use of a define
>>>>>>     eal: new function to create control threads
>>>>>>     eal: set name when creating a control thread
>>>>>>     eal: set affinity for control threads
>>>>>>     examples: use new API to create control threads
>>>>>>
>>>>>>    drivers/net/kni/Makefile                     |  1 +
>>>>>>    drivers/net/kni/rte_eth_kni.c                |  3 +-
>>>>>>    examples/tep_termination/main.c              | 16 +++----
>>>>>>    examples/vhost/main.c                        | 19 +++-----
>>>>>>    lib/librte_eal/bsdapp/eal/eal.c              |  4 +-
>>>>>>    lib/librte_eal/bsdapp/eal/eal_thread.c       |  2 +-
>>>>>>    lib/librte_eal/common/eal_common_proc.c      | 15 ++----
>>>>>>    lib/librte_eal/common/eal_common_thread.c    | 72
>>>>> ++++++++++++++++++++++++++++
>>>>>> lib/librte_eal/common/include/rte_lcore.h    | 26 ++++++++++
>>>>>>    lib/librte_eal/linuxapp/eal/eal.c            |  4 +-
>>>>>>    lib/librte_eal/linuxapp/eal/eal_interrupts.c | 17 ++-----
>>>>>>    lib/librte_eal/linuxapp/eal/eal_thread.c     |  2 +-
>>>>>>    lib/librte_eal/linuxapp/eal/eal_timer.c      | 12 +----
>>>>>>    lib/librte_eal/rte_eal_version.map           |  1 +
>>>>>>    lib/librte_vhost/socket.c                    | 25 ++--------
>>>>>>    15 files changed, 135 insertions(+), 84 deletions(-)
>>>>>>
>>>>> I face a deadlock issue with your series, that Jianfeng patch does not
>>>>> resolve ("eal: fix threads block on barrier"). Reverting the series 
>>>>> and
>>>>> Jianfeng patch makes the issue to disappear.
>>>>>
>>>>> I face the problem in a VM (not seen on the host):
>>>>> # ./install/bin/testpmd -l 0,1,2 --socket-mem 1024 -n 4 --proc-type
>>>>> auto
>>>>> --file-prefix pg -- --portmask=3 --forward-mode=macswap
>>>>> --port-topology=chained --disable-rss -i --rxq=1 --txq=1 --rxd=256
>>>>> --txd=256 --nb-cores=2 --auto-start
>>>>> EAL: Detected 3 lcore(s)
>>>>> EAL: Detected 1 NUMA nodes
>>>>> EAL: Auto-detected process type: PRIMARY
>>>>> EAL: Multi-process socket /var/run/.pg_unix
>>>>>
>>>>>
>>>>> Then it is stuck. Attaching with GDB, I get below backtrace
>>>>> information:
>>>>>
>>>>> (gdb) info threads
>>>>>    Id   Target Id         Frame
>>>>>    3    Thread 0x7f63e1f9f700 (LWP 8808) "rte_mp_handle"
>>>>> 0x00007f63e2591bfd in recvmsg () at
>>>>> ../sysdeps/unix/syscall-template.S:81
>>>>>    2    Thread 0x7f63e179e700 (LWP 8809) "rte_mp_async"
>>>>> pthread_barrier_wait () at
>>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S:71
>>>>> * 1    Thread 0x7f63e32cec00 (LWP 8807) "testpmd" pthread_barrier_wait
>>>>> () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S:71
>>>>> (gdb) bt full
>>>>> #0  pthread_barrier_wait () at
>>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S:71
>>>>> No locals.
>>>>> #1  0x0000000000520c54 in rte_ctrl_thread_create
>>>>> (thread=thread at entry=0x7ffe5c895020, name=name at entry=0x869d86
>>>>> "rte_mp_async", attr=attr at entry=0x0,
>>>>> start_routine=start_routine at entry=0x521030 <async_reply_handle>,
>>>>> arg=arg at entry=0x0)
>>>>>      at /root/src/dpdk/lib/librte_eal/common/eal_common_thread.c:207
>>>>>          params = 0x17b1e40
>>>>>          lcore_id = <optimized out>
>>>>>          cpuset = {__bits = {1, 0 <repeats 15 times>}}
>>>>>          cpu_found = <optimized out>
>>>>>          ret = 0
>>>>> #2  0x00000000005220b6 in rte_mp_channel_init () at
>>>>> /root/src/dpdk/lib/librte_eal/common/eal_common_proc.c:674
>>>>>         path = "/var/run\000.pg_unix_*", '\000' <repeats 1301 
>>>>> times>...
>>>>>          dir_fd = 4
>>>>>          mp_handle_tid = 140066969745152
>>>>>          async_reply_handle_tid = 140066961352448
>>>>> #3  0x000000000050c227 in rte_eal_init (argc=argc at entry=23,
>>>>> argv=argv at entry=0x7ffe5c896378) at
>>>>> /root/src/dpdk/lib/librte_eal/linuxapp/eal/eal.c:775
>>>>>          i = <optimized out>
>>>>>          fctret = 11
>>>>>          ret = <optimized out>
>>>>>          thread_id = 140066989861888
>>>>>          run_once = {cnt = 1}
>>>>>          logid = 0x17b1e00 "testpmd"
>>>>>          cpuset = "T}\211\\\376\177", '\000' <repeats 117 times>,
>>>>> "\020", '\000' <repeats 116 times>...
>>>>>       thread_name = 
>>>>> "X}\211\\\376\177\000\000\226\301\036\342c\177\000"
>>>>>          __func__ = "rte_eal_init"
>>>>> #4  0x0000000000473214 in main (argc=23, argv=0x7ffe5c896378) at
>>>>> /root/src/dpdk/app/test-pmd/testpmd.c:2597
>>>>>          diag = <optimized out>
>>>>>          port_id = <optimized out>
>>>>>          ret = <optimized out>
>>>>>          __func__ = "main"
>>>>> (gdb) thread 2
>>>>> [Switching to thread 2 (Thread 0x7f63e179e700 (LWP 8809))]
>>>>> #0  pthread_barrier_wait () at
>>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S:71
>>>>> 71        cmpl    %edx, (%rdi)
>>>>> (gdb) bt full
>>>>> #0  pthread_barrier_wait () at
>>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S:71
>>>>> No locals.
>>>>> #1  0x0000000000520777 in rte_thread_init (arg=<optimized out>) at
>>>>> /root/src/dpdk/lib/librte_eal/common/eal_common_thread.c:156
>>>>>          params = <optimized out>
>>>>>          start_routine = 0x521030 <async_reply_handle>
>>>>>          routine_arg = 0x0
>>>>> #2  0x00007f63e258add5 in start_thread (arg=0x7f63e179e700) at
>>>>> pthread_create.c:308
>>>>>          __res = <optimized out>
>>>>>          pd = 0x7f63e179e700
>>>>>          now = <optimized out>
>>>>>          unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140066961352448,
>>>>> 1212869169857371576, 0, 8392704, 0, 140066961352448,
>>>>> -1291626103561052744, -1291619793368703560}, mask_was_saved = 0}}, 
>>>>> priv
>>>>>
>>>>> = {pad = {0x0, 0x0, 0x0, 0x0}, data = {
>>>>>                prev = 0x0, cleanup = 0x0, canceltype = 0}}}
>>>>>          not_first_call = <optimized out>
>>>>>          pagesize_m1 = <optimized out>
>>>>>          sp = <optimized out>
>>>>>          freesize = <optimized out>
>>>>> #3  0x00007f63e22b4b3d in clone () at
>>>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
>>>>> No locals.
>>>>> (gdb) thread 3
>>>>> [Switching to thread 3 (Thread 0x7f63e1f9f700 (LWP 8808))]
>>>>> #0  0x00007f63e2591bfd in recvmsg () at
>>>>> ../sysdeps/unix/syscall-template.S:81
>>>>> 81    T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
>>>>> (gdb) bt full
>>>>> #0  0x00007f63e2591bfd in recvmsg () at
>>>>> ../sysdeps/unix/syscall-template.S:81
>>>>> No locals.
>>>>> #1  0x000000000052194e in read_msg (s=0x7f63e1f9d3b0, 
>>>>> m=0x7f63e1f9d5a0)
>>>>>
>>>>> at /root/src/dpdk/lib/librte_eal/common/eal_common_proc.c:258
>>>>>          msglen = <optimized out>
>>>>>          control =
>>>>> "\000\000\000\000\000\000\000\000\336~\f\343c\177\000\000\005", '\000'
>>>>> <repeats 23 times>, "\360\371\033\342c\177\000"
>>>>>          cmsg = <optimized out>
>>>>>          iov = {iov_base = 0x7f63e1f9d5a0, iov_len = 332}
>>>>>        msgh = {msg_name = 0x7f63e1f9d3b0, msg_namelen = 110, msg_iov =
>>>>> 0x7f63e1f9d370, msg_iovlen = 1, msg_control = 0x7f63e1f9d380,
>>>>> msg_controllen = 48, msg_flags = 0}
>>>>> #2  mp_handle (arg=<optimized out>) at
>>>>> /root/src/dpdk/lib/librte_eal/common/eal_common_proc.c:346
>>>>>          msg = {type = 0, msg = {name = '\000' <repeats 63 times>,
>>>>> len_param = 0, num_fds = 0, param = '\000' <repeats 20 times>, "\002",
>>>>> '\000' <repeats 234 times>, fds = {0, 0, 0, 0, 0, 0, 0, 0}}}
>>>>>          sa = {sun_family = 55104,
>>>>>            sun_path =
>>>>> "\371\341c\177\000\000\352\372\f\343c\177\000\000\000\000\000\000\000\000\000\000\377\377\377\377\377\377\377\377\000\367\371\341c\177\000\000\030\000\000\000\000\000\000\000p\327\371\341c\177\000\000\000\367\371\341c\177\000\000\000\367\371\341c\177", 
>>>>>
>>>>>
>>>>> '\000' <repeats 34 times>, "\200\037\000\000\377\377"}
>>>>> #3  0x00007f63e258add5 in start_thread (arg=0x7f63e1f9f700) at
>>>>> pthread_create.c:308
>>>>>          __res = <optimized out>
>>>>>          pd = 0x7f63e1f9f700
>>>>>          now = <optimized out>
>>>>>          unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140066969745152,
>>>>> 1212869169857371576, 0, 8392704, 0, 140066969745152,
>>>>> -1291625004586295880, -1291619793368703560}, mask_was_saved = 0}}, 
>>>>> priv
>>>>>
>>>>> = {pad = {0x0, 0x0, 0x0, 0x0}, data = {
>>>>>                prev = 0x0, cleanup = 0x0, canceltype = 0}}}
>>>>>          not_first_call = <optimized out>
>>>>>          pagesize_m1 = <optimized out>
>>>>>          sp = <optimized out>
>>>>>          freesize = <optimized out>
>>>>> #4  0x00007f63e22b4b3d in clone () at
>>>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
>>>>> No locals.
>>>>>
>>>>> I don't have more info for now.
>>>>>
>>>>
>>>> Thanks for the feedback on this issue. I don't see obvious reason 
>>>> for this deadlock yet.
>>>>
>>>> I'll investigate it asap (not tomorrow, but wednesday). In the worst 
>>>> case, we can revert the series if I cannot find the root cause rapidly.
>>>
>>> I might think that the suggestion from Stephen of destroying the 
>>> barrier can help this issue. I'll try to reproduce it and test it 
>>> before sending a patch to fix it.
>>
>> In case you don't reproduce, feel free to send me the patch to test it.
> 
> Below patch can fix another strange sigsegv issue in my VM. Please check 
> if it works for you. I doubt it's use-after-free problem which could 
> lead to different issues in different env. Please have a try.
> 
> 
> diff --git a/lib/librte_eal/common/eal_common_thread.c 
> b/lib/librte_eal/common/eal_common_thread.c
> index de69452..d91b67d 100644
> --- a/lib/librte_eal/common/eal_common_thread.c
> +++ b/lib/librte_eal/common/eal_common_thread.c
> @@ -205,6 +205,7 @@ rte_ctrl_thread_create(pthread_t *thread, const char 
> *name,
>                  goto fail;
> 
>          pthread_barrier_wait(&params->configured);
> +       pthread_barrier_destroy(&params->configured);
>          free(params);

Should destroy also be called in fail case?

> 
>          return 0;
> 
> Thanks,
> Jianfeng
> 
>>
>> Thanks,
>> Maxime
>>
>>> Thanks,
>>> Jianfeng
>>>
>>>>
>>>> Olivier
>>>>
>>>
> 
> 


-- 
Thanks,
Anatoly


More information about the dev mailing list