[dpdk-dev] [PATCH] eal: add option --iso-cmem for external custom memory

[Burakov, Anatoly]

On 18-Oct-19 11:54 AM, Rajesh Ravi wrote:
> + Srinath
> 
> Thanks Anatoly for reviewing this. Please find my reply inline  below:
> 
> [Anatoly] First of all, what is "iso-cmem"? It doesn't seem to have any 
> defined
> meaning nor any relation to any existing functionality, and it's not
> explained anywhere what is "isolated cmem".
> [Rajesh] I 'll correct commit message to include clearer explanation.
> _
> _
> _Context & purpose_
> We 're using this to improve SPDK performance. When NVMe completion 
> queues are allocated from a certain memory range,
> out of order competions completions  are enabled with our PCIe and it 
> improves performance.
> 
> _Usage_
> spdk_env_init()-->(calls rte_eal_init(), and then calls 
> )spdk_env_dpdk_post_init() [file: spdk/lib/env_dpdk/init.c] 
> --> spdk_mem_map_init() 
> [lib/env_dpdk/memory.c]-->map_cmem_virtual_area() [this is our custom 
> function]
> In map_cmem_virtual_area(): we 're mmaping anonymous &  private region 
> and then creating iova table which makes iova addresses which fall in  
> custom memory region.
> Then we call rte_malloc_heap_memory_add() and then allocate NVMe 
> completion queues from this heap.
> 
> [Anatoly] More importantly, why is this necessary? Type1 map only bypasses
> external segments when adding memory at startup - it doesn't stop you
> from calling rte_vfio_dma_map() to map the memory with VFIO when you
> create the segment.
> [Rajesh] Please correct me if I'm wrong or  missing some thing here.
>    A) We wanted to create a heap from which we can allocate dynamically
> 
>    B) I see that type1_map()  
> [file:dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c] getting called once 
> rte_malloc_heap_memory_add()() was invoked.
>         SPDK uses type1 dma map [spdk/lib/env_dpdk/memory.c]
>          vfio_type1_dma_map() is registered for .dma_map_func member 
> [dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c]
>          So type1_map is called. Not sure whether we can change this.
> 

I'm not sure i'm following. You mean you *don't* want this mapping to be 
performed?

-- 
Thanks,
Anatoly