[dpdk-dev] [PATCH v12 0/3] mempool: add external mempool manager

Jan Viktorin viktorin at rehivetech.com
Wed Jun 15 12:13:58 CEST 2016


Hi,

I've got one last question. Initially, I was interested in creating
my own external memory provider based on a Linux Kernel driver.
So, I've got an opened file descriptor that points to a device which
can mmap a memory regions for me.

...
int fd = open("/dev/uio0" ...);
...
rte_mempool *pool = rte_mempool_create_empty(...);
rte_mempool_set_ops_byname(pool, "uio_allocator_ops");

I am not sure how to pass the file descriptor pointer. I thought it would
be possible by the rte_mempool_alloc but it's not... Is it possible
to solve this case?

The allocator is device-specific.

Regards
Jan

On Wed, 15 Jun 2016 08:47:01 +0100
David Hunt <david.hunt at intel.com> wrote:

> Here's the latest version of the External Mempool Manager patchset.
> It's re-based on top of the latest head as of 14/6/2016, including
> Olivier's 35-part patch series on mempool re-org [1]
> 
> [1] http://dpdk.org/ml/archives/dev/2016-May/039229.html
> 
> v12 changes:
> 
>  * Fixed a comment (function pram h -> ops)
>  * Fixed a typo in mempool docs (callbacki)
> 
> v11 changes:
> 
>  * Fixed comments (added '.' where needed for consistency)
>  * removed ABI breakage notice for mempool manager in deprecation.rst
>  * Added description of the external mempool manager functionality to
>    doc/guides/prog_guide/mempool_lib.rst (John Mc reviewed)
>  * renamed rte_mempool_default.c to rte_mempool_ring.c
> 
> v10 changes:
> 
>  * changed the _put/_get op names to _enqueue/_dequeue to be consistent
>    with the function names
>  * some rte_errno cleanup
>  * comment tweaks about when to set pool_data
>  * removed an un-needed check for ops->alloc == NULL
> 
> v9 changes:
> 
>  * added a check for NULL alloc in rte_mempool_ops_register
>  * rte_mempool_alloc_t now returns int instead of void*
>  * fixed some comment typo's
>  * removed some unneeded typecasts
>  * changed a return NULL to return -EEXIST in rte_mempool_ops_register
>  * fixed rte_mempool_version.map file so builds ok as shared libs
>  * moved flags check from rte_mempool_create_empty to rte_mempool_create
> 
> v8 changes:
> 
>  * merged first three patches in the series into one.
>  * changed parameters to ops callback to all be rte_mempool pointer
>    rather than than pointer to opaque data or uint64.
>  * comment fixes.
>  * fixed parameter to _free function (was inconsistent).
>  * changed MEMPOOL_F_RING_CREATED to MEMPOOL_F_POOL_CREATED
> 
> v7 changes:
> 
>  * Changed rte_mempool_handler_table to rte_mempool_ops_table
>  * Changed hander_idx to ops_index in rte_mempool struct
>  * Reworked comments in rte_mempool.h around ops functions
>  * Changed rte_mempool_hander.c to rte_mempool_ops.c
>  * Changed all functions containing _handler_ to _ops_
>  * Now there is no mention of 'handler' left
>  * Other small changes out of review of mailing list
> 
> v6 changes:
> 
>  * Moved the flags handling from rte_mempool_create_empty to
>    rte_mempool_create, as it's only there for backward compatibility
>  * Various comment additions and cleanup
>  * Renamed rte_mempool_handler to rte_mempool_ops
>  * Added a union for *pool and u64 pool_id in struct rte_mempool
>  * split the original patch into a few parts for easier review.
>  * rename functions with _ext_ to _ops_.
>  * addressed review comments
>  * renamed put and get functions to enqueue and dequeue
>  * changed occurences of rte_mempool_ops to const, as they
>    contain function pointers (security)
>  * split out the default external mempool handler into a separate
>    patch for easier review
> 
> v5 changes:
>  * rebasing, as it is dependent on another patch series [1]
> 
> v4 changes (Olivier Matz):
>  * remove the rte_mempool_create_ext() function. To change the handler, the
>    user has to do the following:
>    - mp = rte_mempool_create_empty()
>    - rte_mempool_set_handler(mp, "my_handler")
>    - rte_mempool_populate_default(mp)
>    This avoids to add another function with more than 10 arguments, duplicating
>    the doxygen comments
>  * change the api of rte_mempool_alloc_t: only the mempool pointer is required
>    as all information is available in it
>  * change the api of rte_mempool_free_t: remove return value
>  * move inline wrapper functions from the .c to the .h (else they won't be
>    inlined). This implies to have one header file (rte_mempool.h), or it
>    would have generate cross dependencies issues.
>  * remove now unused MEMPOOL_F_INT_HANDLER (note: it was misused anyway due
>    to the use of && instead of &)
>  * fix build in debug mode (__MEMPOOL_STAT_ADD(mp, put_pool, n) remaining)
>  * fix build with shared libraries (global handler has to be declared in
>    the .map file)
>  * rationalize #include order
>  * remove unused function rte_mempool_get_handler_name()
>  * rename some structures, fields, functions
>  * remove the static in front of rte_tailq_elem rte_mempool_tailq (comment
>    from Yuanhan)
>  * test the ext mempool handler in the same file than standard mempool tests,
>    avoiding to duplicate the code
>  * rework the custom handler in mempool_test
>  * rework a bit the patch selecting default mbuf pool handler
>  * fix some doxygen comments
> 
> v3 changes:
>  * simplified the file layout, renamed to rte_mempool_handler.[hc]
>  * moved the default handlers into rte_mempool_default.c
>  * moved the example handler out into app/test/test_ext_mempool.c
>  * removed is_mc/is_mp change, slight perf degredation on sp cached operation
>  * removed stack hanler, may re-introduce at a later date
>  * Changes out of code reviews
> 
> v2 changes:
>  * There was a lot of duplicate code between rte_mempool_xmem_create and
>    rte_mempool_create_ext. This has now been refactored and is now
>    hopefully cleaner.
>  * The RTE_NEXT_ABI define is now used to allow building of the library
>    in a format that is compatible with binaries built against previous
>    versions of DPDK.
>  * Changes out of code reviews. Hopefully I've got most of them included.
> 
> The External Mempool Manager is an extension to the mempool API that allows
> users to add and use an external mempool manager, which allows external memory
> subsystems such as external hardware memory management systems and software
> based memory allocators to be used with DPDK.
> 
> The existing API to the internal DPDK mempool manager will remain unchanged
> and will be backward compatible. However, there will be an ABI breakage, as
> the mempool struct is changing. These changes are all contained withing
> RTE_NEXT_ABI defs, and the current or next code can be changed with
> the CONFIG_RTE_NEXT_ABI config setting
> 
> There are two aspects to external mempool manager.
>   1. Adding the code for your new mempool operations (ops). This is
>      achieved by adding a new mempool ops source file into the
>      librte_mempool library, and using the REGISTER_MEMPOOL_OPS macro.
>   2. Using the new API to call rte_mempool_create_empty and
>      rte_mempool_set_ops_byname to create a new mempool
>      using the name parameter to identify which ops to use.
> 
> New API calls added
>  1. A new rte_mempool_create_empty() function
>  2. rte_mempool_set_ops_byname() which sets the mempool's ops (functions)
>  3. An rte_mempool_populate_default() and rte_mempool_populate_anon() functions
>     which populates the mempool using the relevant ops
> 
> Several external mempool managers may be used in the same application. A new
> mempool can then be created by using the new rte_mempool_create_empty function,
> then calling rte_mempool_set_ops_byname to point the mempool to the relevant
> mempool manager callback structure.
> 
> Legacy applications will continue to use the old rte_mempool_create API call,
> which uses a ring based mempool manager by default. These applications
> will need to be modified to use a new external mempool manager.
> 
> The external mempool manager needs to provide the following functions.
>  1. alloc     - allocates the mempool memory, and adds each object onto a ring
>  2. enqueue   - puts an object back into the mempool once an application has
>                 finished with it
>  3. dequeue   - gets an object from the mempool for use by the application
>  4. get_count - gets the number of available objects in the mempool
>  5. free      - frees the mempool memory
> 
> Every time an enqueue/dequeue/get_count is called from the application/PMD,
> the callback for that mempool is called. These functions are in the fastpath,
> and any unoptimised ops may limit performance.
> 
> The new APIs are as follows:
> 
> 1. rte_mempool_create_empty
> 
> struct rte_mempool *
> rte_mempool_create_empty(const char *name, unsigned n, unsigned elt_size,
>     unsigned cache_size, unsigned private_data_size,
>     int socket_id, unsigned flags);
> 
> 2. rte_mempool_set_ops_byname()
> 
> int
> rte_mempool_set_ops_byname(struct rte_mempool *mp, const char *name);
> 
> 3. rte_mempool_populate_default()
> 
> int rte_mempool_populate_default(struct rte_mempool *mp);
> 
> 4. rte_mempool_populate_anon()
> 
> int rte_mempool_populate_anon(struct rte_mempool *mp);
> 
> Please see rte_mempool.h for further information on the parameters.
> 
> 
> The important thing to note is that the mempool ops struct is passed by name
> to rte_mempool_set_ops_byname, which looks through the ops struct array to
> get the ops_index, which is then stored in the rte_memool structure. This
> allow multiple processes to use the same mempool, as the function pointers
> are accessed via ops index.
> 
> The mempool ops structure contains callbacks to the implementation of
> the ops function, and is set up for registration as follows:
> 
> static const struct rte_mempool_ops ops_sp_mc = {
>     .name = "ring_sp_mc",
>     .alloc = rte_mempool_common_ring_alloc,
>     .enqueue = common_ring_sp_enqueue,
>     .dequeue = common_ring_mc_dequeue,
>     .get_count = common_ring_get_count,
>     .free = common_ring_free,
> };
> 
> And then the following macro will register the ops in the array of ops
> structures
> 
> REGISTER_MEMPOOL_OPS(ops_mp_mc);
> 
> For an example of API usage, please see app/test/test_mempool.c, which
> implements a rudimentary "custom_handler" mempool manager using simple mallocs
> for each mempool object. This file also contains the callbacks and self
> registration for the new handler.
> 
> David Hunt (2):
>   mempool: support external mempool operations
>   mbuf: make default mempool ops configurable at build
> 
> Olivier Matz (1):
>   app/test: test external mempool manager
> 
> 



-- 
   Jan Viktorin                  E-mail: Viktorin at RehiveTech.com
   System Architect              Web:    www.RehiveTech.com
   RehiveTech
   Brno, Czech Republic


More information about the dev mailing list