Shared memory between two primary DPDK processes

[Antonio Di Bacco]

Dear Dmitry,

I tried to follow this approach and if I allocate 1GB on primary
process number 1, then I can mmap that memory on the primary process
number 2.
I also tried to convert the virt addr of the allocation made in
primary 1 to phys and then I converted the virt addr returned by mmap
in primary 2 and I got the same phys addr.

Unfortunately, if I try to allocated only 10 MB for example in primary
1, then mmap in primary 2 succeeds but it seems that this virt addr
doesn't correspond to the same phys memory as in primary 1.

In the primary 2, the mmap is used like this:

    int flags = MAP_SHARED | MAP_HUGETLB ;

    uint64_t* addr = (uint64_t*) mmap(NULL, sz, PROT_READ|PROT_WRITE,
flags, my_mem_fd, off);

On Fri, Apr 8, 2022 at 3:26 PM Dmitry Kozlyuk <dmitry.kozliuk at gmail.com> wrote:
>
> 2022-04-08 14:31 (UTC+0200), Antonio Di Bacco:
> > I know that it is possible to share memory between a primary and secondary
> > process using rte_memzone_reserve_aligned to allocate memory in primary
> > that is "seen" also by the secondary. If we have two primary processes
> > (started with different file-prefix) the same approach is not feasible. I
> > wonder how to share a chunk of memory hosted on a hugepage between two
> > primaries.
> >
> > Regards.
>
> Hi Antonio,
>
> Correction: all hugepages allocated by DPDK are shared
> between primary and secondary processes, not only memzones.
>
> I assume we're talking about processes within one host,
> because your previous similar question was about sharing memory between hosts
> (as we have discussed offline), which is out of scope for DPDK.
>
> As for the question directly, you need to map the same part of the same file
> in the second primary as the hugepage is mapped from in the first primary.
> I don't recommend to work with file paths, because their management
> is not straightforward (--single-file-segments, for one) and is undocumented.
>
> There is a way to share DPDK memory segment file descriptors.
> Although public, this DPDK API is dangerous in the sense that you must
> clearly understand what you're doing and how DPDK works.
> Hence the question: what is the task you need this sharing for?
> Maybe there is a simpler way.
>
> 1. In the first primary:
>
>         mz = rte_memzone_reserve()
>         ms = rte_mem_virt2memseg(mz->addr)
>         fd = rte_memseg_get_fd(ms)
>         offset = rte_memseg_get_fd_offset(ms)
>
> 2. Use Unix domain sockets with SCM_RIGHTS
>    to send "fd" and "offset" to the second primary.
>
> 3. In the second primary, after receiving "fd" and "offset":
>
>         flags = MAP_SHARED | MAP_HUGE | (30 << MAP_HUGE_SHIFT)
>         addr = mmap(fd, offset, flags)
>
> Note that "mz" may consist of multiple "ms" depending on the sizes
> of the zone and hugepages, and on the zone alignment.
> Also "addr" may (and probably will) differ from "mz->addr".
> It is possible to pass "mz->addr" and try to force it,
> like DPDK does for primary/secondary.
>