[dpdk-dev] [RFC]app/testpmd: time-consuming question of mlockall function execution

David Marchand david.marchand at redhat.com
Mon Feb 24 09:48:08 CET 2020


Hello,

On Mon, Feb 24, 2020 at 7:35 AM humin (Q) <humin29 at huawei.com> wrote:
> We found that if OS transparent hugepage uses non 'always', mlockall function in the main function of testpmd takes more than 25s to execute.
> The results of running on both x86 and ARM are the same. It's very unreasonable and deadly. The enable status of transparent hugepage on OS can be viewed by the following command.
> [root at X6000-C23-1U dpdk]#cat /sys/kernel/mm/transparent_hugepage/enabled
> always [madvise] never
>
> Transparent hugepage on ARM is configured as 'madvise', 'never' or 'always', testpmd runs with using strace as follows:
> ******************************* Transparent hugepage is configured as 'madvise'  ******************************* [root at X6000-C23-1U dpdk]# strace -T -e trace=mlockall ./testpmd -l 1-4 -w 0000:7d:01.0 --iova-mode=va -- -i
[snip]
> Interactive-mode selected
> mlockall(MCL_CURRENT|MCL_FUTURE)        = 0 <25.736362>
> <---------------------- Hang for 25 seconds
[snip]
>
> *****************************  Transparent hugepage is configured as 'never'  ********************************* [root at X6000-C23-1U dpdk]# strace -T -e trace=mlockall ./testpmd -l 1-4 -w 0000:7d:01.0 --iova-mode=va -- -i
[snip]
> Interactive-mode selected
> mlockall(MCL_CURRENT|MCL_FUTURE)        = 0 <25.737757>
> <---------------------- Hang for 25 seconds
[snip]
>
> *****************************  Transparent hugepage is configured as 'always'  ********************************* [root at X6000-C23-1U dpdk]# strace -T -e trace=mlockall testpmd -l 1-4 -w
> 0000:7d:01.0 --iova-mode=va -- -i
> strace: Can't stat 'testpmd': No such file or directory [root at X6000-C23-1U dpdk]# strace -T -e trace=mlockall ./testpmd -l 1-4 -w 0000:7d:01.0 --iova-mode=va -- -i
[snip]
> Interactive-mode selected
> mlockall(MCL_CURRENT|MCL_FUTURE)        = 0 <0.208571>
> <---------------------- No Hang
[snip]

>
> We have also seen some discussions on this issue in following page:
> https://bugzilla.redhat.com/show_bug.cgi?id=1786923
>
> David Marchand has a related patch, as following page:
> https://github.com/david-marchand/dpdk/commit/f9e1b9fa101c9f4f16c0717401a55790aecc6484
> but this patch doesn't seem to have been submitted to the community.

Yes, this is not ready, I worked on this locally since then (and the
last version is not pushed to my github).
The main problem is that I have not been able to reproduce Eelco issue
so far which justified the addition of mlockall().


> Transparent hugepage on ARM is configured as 'madvise' or 'never', testpmd runs with using strace as follows:
> *******************************************************
> [root at X6000-C23-1U app]# strace -T -e trace=mlockall ./testpmd -l 1-4 -w
> 0000:7d:01.0 --iova-mode=va -- -i
[snip]
> Interactive-mode selected
> mlockall(MCL_CURRENT|MCL_FUTURE|MCL_ONFAULT) = 0 <1.955947>
> <---------------------- Hang for less than 2 seconds
> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc Done
> testpmd> quit
>
> Bye...
> +++ exited with 0 +++
>
> We'd like to know what is the current development on this issue in dpdk community. Thanks

There is also a report about coredumps being huge.
On this topic, there might be something to do with madvise + MADV_DONTDUMP.


I see various options from the hardest to the easiest:
- drop multiprocess support,
- rework the memory allocator so that this kind of side effect
(mapping a huge number of unused pages) is not hit,
- change testpmd so that it locks only the pages of interest (this is
the patch that I had been looking at for now),
- change testpmd so that it does not call mlockall by default,

The last option is likely the quickest workaround.

I write "options", but the last two points feel like "band aid" fixes.
And it does not solve the problem for other applications that will
have to implement similar workarounds.

Anatoly warned that touching the memory allocator is going to be hell.
Quite sad to reach this state because it feels like people are just
starting to hit the changes that entered dpdk 18.05.


Of course, other ideas welcome!

--
David Marchand



More information about the dev mailing list