[dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Kumar, Ravi1]

>On 22-Jun-18 5:37 PM, Kumar, Ravi1 wrote:
>> Hi,
>> 
>> As the memory subsystem in DPDK 18.05 is reworked, it has introduced a problem for AMD EPYC 2P platforms.
>> The issue is that DPDK 18.05 only works with up to 4 NUMAs. For AMD EPYC 2P platforms, DPDK now only works with P0 (NUMA 0-3) and does not work with P1 (NUMA 4-7).
>> 
>> The problem can be fixed by reducing some of the default settings of the memory subsystem.
>> 
>> To solve this issue:
>> -              We can create our own config file for our integrated 10G NIC, that is for amd_xgbe PMD. This will make amd_xgbe immune to this problem.
>> -              However, when any other NIC (Intel, Mellanox, Cavium or Broadcom etc.) is plugged into NUMA 4-7, the problem will still be exposed.
>> -              If we only fix it in "config/common_base", it will cover all cases.
>> 
>> Our current workaround is:
>> Edit config file "./config/common_base" and change the following line
>>                  CONFIG_RTE_MAX_MEM_MB_PER_TYPE=131072
>> TO
>>                  CONFIG_RTE_MAX_MEM_MB_PER_TYPE=65536
>> 
>> Any better solution for this issue is welcome.
>> 
>> We would appreciate if this issue can be fixed in the next release (18.08) so the STOCK version of DPDK works on AMD EPYC 2P platforms.
>> 
>> Regards,
>> Ravi
>> 
>
>Hi Ravi,
>
>What is the reason behind this limitation? Is it too much virtual memory being preallocated?
>
>--
>Thanks,
>Anatoly
>
Hi Anatoly,

We believe this is true.  By default, too much virtual memory is being preallocated. The result is it can only support up to 4 NUMAs.

Our workaround is to reduce the amount of preallocated virtual memory by half, so to support up to 8 NUMAs.

Regards,
Ravi