[dpdk-dev] [RFC] net/mlx5: add reclaim_mem_mode devarg

[Suanming Mou]

Currently, when flow destroyed, some memory resources may still be kept 
as cached to help next time create flow more efficiently.

Some system may need the resources to be more flexible with flow create 
and destroy.  After peak time, with millions of flows destroyed, the 
system would prefer the resources to be reclaimed completely, no cache 
is needed. Then the resources can be allocated and used by other 
components. The system is not so sensitive about the flow insertion 
rate, but more care about the resources.

Both DPDK mlx5 PMD driver and the low level component rdma-core have 
provided the flow resources to be configured cached or not, but there is 
no APIs or parameters exposed to user to configure the flow resources 
cache mode. In this case, introduce a new PMD devarg to let user 
configure the flow resources cache mode will be helpful.

The intention is to add a new "reclaim_mem_mode" to help user configure 
if the destroyed flows' cache resources should be kept or not.
Their will be three mode can be chosen:
1. 0(none). It means the flow resources will be cached as usual. The 
resources will be cached, helpful with flow insertion rate.
2. 1(light). It will only enable the DPDK PMD level resources reclaim.
3. 2(aggressive). Both DPDK PMD level and rdma-core low level will be 
configured as reclaimed mode.

With these three mode, user can configure the resources cache mode with 
different level.
The sample configuration as below:
./testpmd -n 4  -w 0000:d8:00.0,reclaim_mem_mode=2 -- --burst=64 
--mbcache=512 -i  --nb-cores=7

Thanks,
SuanmingMou