Failure while allocating 1GB hugepages

[Antonio Di Bacco]

Hi,
I have the same behaviour with the code in this message.

The first rte_memzone_reserve_aligned() call requesting 1.5GB
contiguous memory always fails, while the second one is always
successful.

It seems in eal_memalloc_is_contig() the 'msl->memseg_arr' items are inverted:
when there is the sequence FC0000000, F80000000 the allocation fails,
while the segments sequence F80000000, FC0000000 is fine.
>From my understaning 'msl->memseg_arr' comes from
'rte_eal_get_configuration()->mem_config;' which is rte_config
declared in eal_common_config.c

Is there an explanation for this swinging behaviour ?

Br,


Here the source code:

#include <stdio.h>
#include <rte_eal.h>
#include <rte_memzone.h>
#include <rte_errno.h>
#include <unistd.h>

int main(int argc, char **argv)
{
const struct rte_memzone *mz;

if (rte_eal_init(argc, argv) < 0)
return -1;

printf("Allocating : 1.5GB\n");

mz = rte_memzone_reserve_aligned("my_huge_mem",  0x60000000,
rte_socket_id(), RTE_MEMZONE_1GB | RTE_MEMZONE_IOVA_CONTIG,
RTE_CACHE_LINE_SIZE);
if (mz == NULL)
{
printf("  Fail(1): errno %s\n", rte_strerror(rte_errno));
mz = rte_memzone_reserve_aligned("my_huge_mem",  0x60000000,
rte_socket_id(), RTE_MEMZONE_1GB | RTE_MEMZONE_IOVA_CONTIG,
RTE_CACHE_LINE_SIZE);
if (mz == NULL)
{
printf("  Fail(2): errno %s\n", rte_strerror(rte_errno));
return -2;
}
}

printf("  Success: phy[%p] size[%zu]\n", mz->iova, mz->len);
rte_memzone_free(mz);
rte_eal_cleanup();
return 0;
}

I added two RTE_LOG notices in eal_memalloc_is_contig @
eal_common_memalloc.c around line 324, after rte_fbarray_get() to
print ms->iova

/* skip first iteration */
ms = rte_fbarray_get(&msl->memseg_arr, start_seg);
RTE_LOG(NOTICE, EAL, "memseg_arr[0] = %lX \n", ms->iova); // DEBUG
cur = ms->iova;
expected = cur + pgsz;

/* if we can't access IOVA addresses, assume non-contiguous */
if (cur == RTE_BAD_IOVA)
return false;

for (cur_seg = start_seg + 1; cur_seg < end_seg;
cur_seg++, expected += pgsz) {
ms = rte_fbarray_get(&msl->memseg_arr, cur_seg);
RTE_LOG(NOTICE, EAL, "memseg_arr[%d] = %lX \n", cur_seg, ms->iova); // DEBUG

if (ms->iova != expected)
return false;
}

The output is:

Allocating : 1.5GB
EAL: memseg_arr[0] = FC0000000
EAL: memseg_arr[1] = F80000000
  Fail(1): errno Cannot allocate memory
EAL: memseg_arr[0] = F80000000
EAL: memseg_arr[1] = FC0000000
EAL: memseg_arr[0] = F80000000
EAL: memseg_arr[1] = FC0000000
EAL: memseg_arr[0] = F80000000
EAL: memseg_arr[1] = FC0000000
EAL: memseg_arr[0] = F80000000
EAL: memseg_arr[1] = FC0000000
  Success: phy[0xfa0000000] size[1610612736]

On Thu, May 30, 2024 at 5:00 PM Dmitry Kozlyuk <dmitry.kozliuk at gmail.com> wrote:
>
> 2024-05-30 12:28 (UTC+0200), Antonio Di Bacco:
> > Just in case I need, let us say, 1.5 GB CONTIGUOUS memory zone,
> > would it be fine to use something like this as GRUB config in Linux?
> >
> > default_hugepagesz=2G hugepagesz=2G hugepages=4"
>
> On x86, "hugepagesz" and "default_hugepagesz" may be either 2M or 1G.
> There is no way to *guarantee* that there will be
> two physically adjacent 1G hugepages forming 1.5GB contiguous space,
> but in practice these options, with the above correction, will do.
>
> Note that by default the kernel will spread hugepages between NUMA nodes.
> You can control this by a more elaborate form of "hugepages" option:
>
>         https://docs.kernel.org/admin-guide/mm/hugetlbpage.html