[dpdk-dev] [PATCH v2] mem: accelerate dpdk program startup by reuse page from page cache

[Burakov, Anatoly]

On 09-Nov-18 9:23 AM, jianmingfan wrote:
> --- fix coding style of the previous patch
> 
> During procless startup, dpdk invokes clear_hugedir() to unlink all
> hugepage files under /dev/hugepages. Then in map_all_hugepages(),
> it invokes mmap to allocate and zero all the huge pages as configured
> in /sys/kernel/mm/hugepages/xxx/nr_hugepages.
> 
> This cause startup process extreamly slow with large size of huge page
> configured.
> 
> In our use case, we usually configure as large as 200GB hugepages in our
> router. It takes more than 50s each time dpdk process startup to clear
> the pages.
> 
> To address this issue, user can turn on --reuse-map switch. With it,
> dpdk will check the validity of the exiting page cache under
> /dev/hugespages. If valid, the cache will be reused not deleted,
> so that the os doesn't need to zero the pages again.
> 
> However, as there are a lot of users ,e.g. rte_kni_alloc, rely on the
> os zeor page behavior. To keep things work, I add memset during
> malloc_heap_alloc(). This makes sense due to the following reason.
> 1) user often configure hugepage size too large to be used by the program.
> In our router, 200GB is configured, but less than 2GB is actually used.
> 2) dpdk users don't call heap allocation in performance-critical path.
> They alloc memory during process bootup.
> 
> Signed-off-by: Jianming Fan <fanjianming at jd.com>
> ---

I believe this issue is better solved by actually fixing all of the 
memory that DPDK leaves behind. We already have rte_eal_cleanup() call 
which will deallocate any EAL-allocated memory that have been reserved, 
and an exited application should free any memory it was using so that 
memory subsystem could free it back to the system, thereby not needing 
any cleaning of hugepages at startup.

If your application does not e.g. free its mempools on exit, it should 
:) Chances are, the problem will go away. The only circumstance where 
this may not work is if you preallocated your memory using 
-m/--socket-mem flag.

-- 
Thanks,
Anatoly