[dpdk-dev] [PATCH] mem: add --match-allocations
Burakov, Anatoly
anatoly.burakov at intel.com
Fri Dec 14 10:20:23 CET 2018
On 13-Dec-18 9:55 PM, Jim Harris wrote:
> SPDK uses the rte_mem_event_callback_register API to
> create RDMA memory regions (MRs) for newly allocated regions
> of memory. This is used in both the SPDK NVMe-oF target
> and the NVMe-oF host driver.
Hi Jim,
Here and in other places - it appears that you're adding double spaces
after end of sentence. We generally don't do that in DPDK.
>
> DPDK creates internal malloc_elem structures for these
> allocated regions. As users malloc and free memory, DPDK
> will sometimes merge malloc_elems that originated from
> different allocations that were notified through the
> registered mem_event callback routine. This results
> in subsequent allocations that can span across multiple
> RDMA MRs. This requires SPDK to check each DPDK buffer to
> see if it crosses an MR boundary, and if so, would have to
> add considerable logic and complexity to describe that
> buffer before it can be accessed by the RNIC. It is somewhat
> analagous to rte_malloc returning a buffer that is not
> IOVA-contiguous.
>
> As a malloc_elem gets split and some of these elements
> get freed, it can also result in DPDK sending an
> RTE_MEM_EVENT_FREE notification for a subset of the
> original RTE_MEM_EVENT_ALLOC notification. This is also
> problematic for RDMA memory regions, since unregistering
> the memory region is all-or-nothing. It is not possible
> to unregister part of a memory region.
>
> To support these types of applications, this patch adds
> a new --match-allocations EAL init flag. When this
> flag is specified, malloc elements from different
> hugepage allocations will never be merged. Memory will
> also only be freed back to the system (with the requisite
> memory event callback) exactly as it was originally
> allocated.
>
> Since part of this patch is extending the size of struct
> malloc_elem, we also fix up the malloc autotests so they
> do not assume its size exactly fits in one cacheline.
>
> Signed-off-by: Jim Harris <james.r.harris at intel.com>
> ---
> doc/guides/prog_guide/env_abstraction_layer.rst | 13 +++++++++++++
> lib/librte_eal/common/eal_common_options.c | 6 ++++++
> lib/librte_eal/common/eal_internal_cfg.h | 1 +
> lib/librte_eal/common/eal_options.h | 2 ++
> lib/librte_eal/common/malloc_elem.c | 16 ++++++++++++----
> lib/librte_eal/common/malloc_elem.h | 6 +++++-
> lib/librte_eal/common/malloc_heap.c | 13 ++++++++++++-
> lib/librte_eal/linuxapp/eal/eal.c | 5 +++++
> test/test/test_malloc.c | 25 +++++++++++++++++++------
> 9 files changed, 75 insertions(+), 12 deletions(-)
>
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 8b5d050c7..c4032161f 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -169,6 +169,19 @@ not allow acquiring or releasing hugepages from the system at runtime.
> If neither ``-m`` nor ``--socket-mem`` were specified, the entire available
> hugepage memory will be preallocated.
>
> ++ Hugepage allocation matching
> +
> +This behavior is enabled by specifying the ``--match-allocations`` command-line
> +switch to the EAL. This switch is mutually exclusive with ``--legacy-mem``.
I wouldn't say it's "mutually exclusive" as much as it's simply
unsupported because there's no dynamic memory allocation in legacy mode.
(it's also unsupported with --no-huge as well). It is also a Linux-only
flag.
Please also update doc/guides/linux_gsg/linux_eal_parameters.rst to have
this parameter. It would also be a good idea to add a note to
doc/guides/rel_notes/release_19.02.rst.
> +
> +Some applications using memory event callbacks may require that hugepages be
> +freed exactly as they were allocated. These applications may also require
> +that any allocation from the malloc heap not span across allocations
> +associated with two different memory event callbacks. Hugepage allocation
> +matching can be used by these types of applications to satisfy both of these
> +requirements. This can result in some increased memory usage which is
> +very dependent on the memory allocation patterns of the application.
> +
> + 32-bit support
>
> Additional restrictions are present when running in 32-bit mode. In dynamic
> diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
> index e31eca5c0..034447929 100644
> --- a/lib/librte_eal/common/eal_common_options.c
> +++ b/lib/librte_eal/common/eal_common_options.c
> @@ -79,6 +79,7 @@ eal_long_options[] = {
> {OPT_VMWARE_TSC_MAP, 0, NULL, OPT_VMWARE_TSC_MAP_NUM },
> {OPT_LEGACY_MEM, 0, NULL, OPT_LEGACY_MEM_NUM },
> {OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
> + {OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
> {0, 0, NULL, 0 }
> };
>
> @@ -1424,6 +1425,11 @@ eal_check_common_options(struct internal_config *internal_cfg)
> "with --"OPT_IN_MEMORY"\n");
> return -1;
> }
> + if (internal_cfg->legacy_mem && internal_cfg->match_allocations) {
> + RTE_LOG(ERR, EAL, "Option --"OPT_LEGACY_MEM" is not compatible "
> + "with --"OPT_MATCH_ALLOCATIONS"\n");
> + return -1;
> + }
Probably should add a check for --no-huge as well.
Everything else looks OK to me, so, provided the above comments are
addressed,
Reviewed-by: Anatoly Burakov <anatoly.burakov at intel.com>
--
Thanks,
Anatoly
More information about the dev
mailing list