[dpdk-dev] [PATCH v2] mem: accelerate dpdk program startup by reuse page from page cache

[Burakov, Anatoly]

On 09-Nov-18 12:20 PM, Burakov, Anatoly wrote:
> On 09-Nov-18 9:23 AM, jianmingfan wrote:
>> --- fix coding style of the previous patch
>>
>> During procless startup, dpdk invokes clear_hugedir() to unlink all
>> hugepage files under /dev/hugepages. Then in map_all_hugepages(),
>> it invokes mmap to allocate and zero all the huge pages as configured
>> in /sys/kernel/mm/hugepages/xxx/nr_hugepages.
>>
>> This cause startup process extreamly slow with large size of huge page
>> configured.
>>
>> In our use case, we usually configure as large as 200GB hugepages in our
>> router. It takes more than 50s each time dpdk process startup to clear
>> the pages.
>>
>> To address this issue, user can turn on --reuse-map switch. With it,
>> dpdk will check the validity of the exiting page cache under
>> /dev/hugespages. If valid, the cache will be reused not deleted,
>> so that the os doesn't need to zero the pages again.
>>
>> However, as there are a lot of users ,e.g. rte_kni_alloc, rely on the
>> os zeor page behavior. To keep things work, I add memset during
>> malloc_heap_alloc(). This makes sense due to the following reason.
>> 1) user often configure hugepage size too large to be used by the 
>> program.
>> In our router, 200GB is configured, but less than 2GB is actually used.
>> 2) dpdk users don't call heap allocation in performance-critical path.
>> They alloc memory during process bootup.
>>
>> Signed-off-by: Jianming Fan <fanjianming at jd.com>
>> ---
> 
> I believe this issue is better solved by actually fixing all of the 
> memory that DPDK leaves behind. We already have rte_eal_cleanup() call 
> which will deallocate any EAL-allocated memory that have been reserved, 
> and an exited application should free any memory it was using so that 
> memory subsystem could free it back to the system, thereby not needing 
> any cleaning of hugepages at startup.
> 
> If your application does not e.g. free its mempools on exit, it should 
> :) Chances are, the problem will go away. The only circumstance where 
> this may not work is if you preallocated your memory using 
> -m/--socket-mem flag.
> 

To clarify - all of the above is only applicable to 18.05 and beyond. 
The map_all_hugepages() function only gets called in the legacy mem 
init, so this patch solves a problem that does not exist on recent DPDK 
versions in the first place - faster initialization is one of the key 
reasons why the new memory subsystem was developed.

-- 
Thanks,
Anatoly