Shared memory between two primary DPDK processes

[Dmitry Kozlyuk]

2022-07-07 00:14 (UTC+0200), Antonio Di Bacco:
> Dear Dmitry,
> 
> I tried to follow this approach and if I allocate 1GB on primary
> process number 1, then I can mmap that memory on the primary process
> number 2.
> I also tried to convert the virt addr of the allocation made in
> primary 1 to phys and then I converted the virt addr returned by mmap
> in primary 2 and I got the same phys addr.
> 
> Unfortunately, if I try to allocated only 10 MB for example in primary
> 1, then mmap in primary 2 succeeds but it seems that this virt addr
> doesn't correspond to the same phys memory as in primary 1.
> 
> In the primary 2, the mmap is used like this:
> 
>     int flags = MAP_SHARED | MAP_HUGETLB ;
> 
>     uint64_t* addr = (uint64_t*) mmap(NULL, sz, PROT_READ|PROT_WRITE,
> flags, my_mem_fd, off);

Hi Antonio,

From `man 2 mmap`:

   Huge page (Huge TLB) mappings
       For  mappings that employ huge pages, the requirements for the
       arguments of mmap() and munmap() differ somewhat from the requirements
       for mappings that use the native system page size.

       For mmap(), offset must be a multiple of the underlying huge page
       size.  The system automatically aligns length to be a  multiple  of
       the underlying huge page size.

       For munmap(), addr, and length must both be a multiple of the
       underlying huge page size.

Probably process 1 maps a 1 GB hugepage:
DPDK does so if 1 GB hugepages are used even if you only allocate 10 MB.
You can examine memseg to see what size it is (not memzone!).
Hugepage size is a property of each mounted HugeTBL filesystem.
It determines which kernel pool to use.
Pools are over different sets of physical pages.
This means that the kernel doesn't allow to map given page frames
as 1 GB and 2 MB hugepages at the same time via hugetlbfs.
I'm surprised mmap() works at all in your case
and suspect that it is mapping 2 MB hugepages in process 2.

The solution may be, in process 2:

base_offset = RTE_ALIGN_FLOOR(offset, hugepage_size)
map_addr = mmap(fd, size=hugepage_size, offset=base_offset)
addr = RTE_PTR_ADD(map_addr, offset - base_offset)

Note that if [offset; offset+size) crosses a hugepage boundary,
you have to map more than one page.