DPDK 19.11.3 with multi processes and external physical memory: unable to receive traffic in the secondary processes

[Burakov, Anatoly]

Hi Asaf,

On 12-Jul-22 7:05 AM, Asaf Sinai wrote:
> Hi,
> 
> We run DPDK 19.11.3 with multi processes.
> 
> When using external memory for DPDK (instead of huge pages), we are 
> unable to receive traffic in the secondary processes.
> 
>   * We have external physical memory regions (excluded from Linux):
> 
> /[ULP-NG]# cat /proc/cmdline/
> 
> /console=ttyS0,19200 isolcpus=1-127 smp_affinity=1 root=/dev/ram0 
> rwmode=r net.ifnames=0 biosdevname=0 systemd.show_status=0/
> 
> /memmap=0x1000000!0x60000000 *memmap=0x90000000!0x800000000 
> memmap=0x90000000!0x1800000000*quiet cloud-init=disabled/
> 
>   * We make all DPDK initializations in the primary process, including
>     mapping the mentioned memory regions via “/dev/mem”.
> 
> After these memory mapping, we register the external memory, as 
> described in 
> http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory 
> <http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory>, 
> and allocate memory pools:
> 
> /.../
> 
> //* Map physical to virtual *//
> 
> /int memFd = open("/dev/mem", O_RDWR);/
> 
> /extMemInfo[i].pageSize   = pPagesInfo->maxPageSize;/
> 
> /extMemInfo[i].memRegSize = 0x0000000080000000;/
> 
> /extMemInfo[i].memRegAddr = mmap(NULL, extMemInfo[i].memRegSize, 
> PROT_READ | PROT_WRITE,/
> 
> /                             MAP_SHARED, memFd, memRegPhysAddr[i]);/
> 
> //
> 
> //* Create heap *//
> 
> /sprintf(extMemInfo[i].heapName, "extMemHeapSocket_%u", i);/
> 
> /rv = rte_malloc_heap_create(extMemInfo[i].heapName);/
> 
> //
> 
> //* Save heap socket ID *//
> 
> /rv = rte_malloc_heap_get_socket(extMemInfo[i].heapName);/
> 
> /extMemInfo[i].socketId = rv;/
> 
> //
> 
> //* Add memory region to heap *//
> 
> /rv = rte_malloc_heap_memory_add(extMemInfo[i].heapName, 
> extMemInfo[i].memRegAddr,/
> 
> /                                extMemInfo[i].memRegSize, NULL, 0, 
> extMemInfo[i].pageSize);/
> 
> /.../

Please correct me if I'm wrong, but it seems like you're specifying NULL 
as IOVA table, so that means external memory will not get IOVA addresses 
set. You need to find (probably using rte_mem_virt2phys() function) the 
individual page addresses, put them into an array, and pass it as an 
argument to add_memory call. See documentation[1], particularly the 
iova_addrs[] parameter.

[1] 
http://doc.dpdk.org/api/rte__malloc_8h.html#af6360dea35bdf162feeb2b62cf149fd3

> 
> //* Allocate memory pool *//
> 
> /memPool = rte_mempool_create(poolName, nbufs, DPDK_MBUF_SIZE, 
> poolCacheSize,/
> 
> /                             sizeof(struct rte_pktmbuf_pool_private),/
> 
> /                             rte_pktmbuf_pool_init, NULL,/
> 
> /                             und_pktmbuf_init, NULL, 
> extMemInfo[i].socketId, DPDK_NO_FLAGS);/
> 
> /.../
> 
> Please note, that during calls to “/rte_malloc_heap_memory_add/”, we see 
> the following warnings:
> 
> */EAL: WARNING! Base virtual address hint (0x2101005000 != 
> 0x7ffff4627000) not respected!/*
> 
> /EAL:    This may cause issues with mapping memory into secondary processes/
> 
> */EAL: WARNING! Base virtual address hint (0x210100b000 != 
> 0x7ffff4619000) not respected!/*
> 
> /EAL:    This may cause issues with mapping memory into secondary processes/
> 
> And after executing “/rte_mempool_create/”, physical addresses of the 
> allocated memory zones are bad:

Yes, because you did not specify them as part of 
rte_malloc_heap_add_memory() call :)

> 
> /[Jul 12 12:02:17] [1875] extMemConfig: heapName=extMemHeapSocket_0, 
> socketId=256, memRegAddr=0x7fff58000000, memRegSize=2415919104, 
> pageSize=2097152/
> 
> /[Jul 12 12:02:18] [1875] memZone: name=MP_ndPool_0, socket_id=256, 
> vaddr=0x7fffe7e7bec0-0x7fffe7ffffc0, 
> *paddr=0xffffffffffffffff-0x1840ff*, len=1589504, hugepage_sz=2MB/
> 
>   * In the next step, we spawn the secondary processes from the primary
>     one, using fork().
> 
> This way, all DPDK data and memory mappings are the same on all 
> processes. For example:

This is a recipe for disaster, because DPDK creates internal data 
structures that are mapped as files, and primary process expects to be 
the sole owner and user of those data structures. If you would like to 
create a secondary process using fork() at all, you should fork() first, 
and then call rte_eal_init() in the secondary process. A forked primary 
process is not what we mean when we say "secondary process".

In addition, see documentation[1] for `rte_malloc_heap_memory_add()` - 
there's a note there that states:

Before accessing this memory in other processes, it needs to be attached 
in each of those processes by calling rte_malloc_heap_memory_attach in 
each other process.

So, not only you need to call secondary process EAL init, you will also 
have to call `rte_malloc_heap_memory_attach()` before you can use your 
shared external memory in your secondary process.

Hope this is helpful!

> 
> _Mapping in primary process:_
> 
> /cat /proc/1875/maps/
> 
> /.../
> 
> */7ffec8000000-7fff58000000 rw-s 1800000000 00:06 
> 5                        /dev/mem/*
> 
> */7fff58000000-7fffe8000000 rw-s 800000000 00:06 
> 5                         /dev/mem/*
> 
> /.../
> 
> _Mapping in secondary process:_
> 
> /cat /proc/2676/maps/
> 
> /.../
> 
> */7ffec8000000-7fff58000000 rw-s 1800000000 00:06 
> 5                        /dev/mem/*
> 
> */7fff58000000-7fffe8000000 rw-s 800000000 00:06 
> 5                         /dev/mem/*
> 
> /.../
> 
>   * Unfortunately, when traffic is received by the secondary processes,
>     we see the following printout on the primary process:
> 
> */i40e_dev_alarm_handler(): ICR0: malicious programming detected/*
> 
>   * Additionally, we saw no example of using external physical shared
>     memory on secondary processes.
> 
> DPDK test applications show usage of anonymous private memory on the 
> primary process only.
> 
> What are we missing?
> 
> Can you please advise?
> 
> Thanks,
> 
> Asaf
> 
> Radware
> 


-- 
Thanks,
Anatoly