Shared memory between two primary DPDK processes

[Antonio Di Bacco]

I did a short program where a  primary (--file-prefix=p1) allocates a
memzone and generates a file descriptor that is passed to another primary
(--file-prefix=p2) .
The process P2 tries to mmap the memory but I get an error (Bad file
descriptor):

        uint64_t* addr = mmap(NULL, 1024*1024*1024, PROT_READ, flags,
mem_fd, 0);
        if (addr == -1)
            perror("mmap");

Il giorno ven 8 apr 2022 alle ore 23:08 Antonio Di Bacco <
a.dibacco.ks at gmail.com> ha scritto:

>
>
> Il giorno ven 8 apr 2022 alle ore 15:26 Dmitry Kozlyuk <
> dmitry.kozliuk at gmail.com> ha scritto:
>
>> 2022-04-08 14:31 (UTC+0200), Antonio Di Bacco:
>> > I know that it is possible to share memory between a primary and
>> secondary
>> > process using rte_memzone_reserve_aligned to allocate memory in primary
>> > that is "seen" also by the secondary. If we have two primary processes
>> > (started with different file-prefix) the same approach is not feasible.
>> I
>> > wonder how to share a chunk of memory hosted on a hugepage between two
>> > primaries.
>> >
>> > Regards.
>>
>> Hi Antonio,
>>
>> Correction: all hugepages allocated by DPDK are shared
>> between primary and secondary processes, not only memzones.
>>
>> I assume we're talking about processes within one host,
>> because your previous similar question was about sharing memory between
>> hosts
>> (as we have discussed offline), which is out of scope for DPDK.
>>
>> As for the question directly, you need to map the same part of the same
>> file
>> in the second primary as the hugepage is mapped from in the first primary.
>> I don't recommend to work with file paths, because their management
>> is not straightforward (--single-file-segments, for one) and is
>> undocumented.
>>
>> There is a way to share DPDK memory segment file descriptors.
>> Although public, this DPDK API is dangerous in the sense that you must
>> clearly understand what you're doing and how DPDK works.
>> Hence the question: what is the task you need this sharing for?
>> Maybe there is a simpler way.
>>
>> 1. In the first primary:
>>
>>         mz = rte_memzone_reserve()
>>         ms = rte_mem_virt2memseg(mz->addr)
>>         fd = rte_memseg_get_fd(ms)
>>         offset = rte_memseg_get_fd_offset(ms)
>>
>> 2. Use Unix domain sockets with SCM_RIGHTS
>>    to send "fd" and "offset" to the second primary.
>>
>> 3. In the second primary, after receiving "fd" and "offset":
>>
>>         flags = MAP_SHARED | MAP_HUGE | (30 << MAP_HUGE_SHIFT)
>>         addr = mmap(fd, offset, flags)
>>
>> Note that "mz" may consist of multiple "ms" depending on the sizes
>> of the zone and hugepages, and on the zone alignment.
>> Also "addr" may (and probably will) differ from "mz->addr".
>> It is possible to pass "mz->addr" and try to force it,
>> like DPDK does for primary/secondary.
>>
>
>
> Thank you Dmitry, it is really incredible how deep your knowledge is. I
> will give it a try.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mails.dpdk.org/archives/users/attachments/20220411/c74493ce/attachment.htm>