[dpdk-dev] Question about cache_size in rte_mempool_create

[Roy Shterman]

נשלח מה-iPhone שלי

‫ב-24 בנוב׳ 2017, בשעה 12:03, ‏‏Bruce Richardson ‏<bruce.richardson at intel.com> כתב/ה:‬

>> On Fri, Nov 24, 2017 at 11:39:54AM +0200, roy wrote:
>> Thanks for your answer, but I cannot understand the dimension of the ring
>> and it is affected by the cache size.
>> 
>>> On 24/11/17 11:30, Bruce Richardson wrote:
>>>> On Thu, Nov 23, 2017 at 11:05:11PM +0200, Roy Shterman wrote:
>>>> Hi,
>>>> 
>>>> In the documentation it says that:
>>>> 
>>>>  * @param cache_size
>>>>  *   If cache_size is non-zero, the rte_mempool library will try to
>>>>  *   limit the accesses to the common lockless pool, by maintaining a
>>>>  *   per-lcore object cache. This argument must be lower or equal to
>>>>  *   CONFIG_RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.* It is advised to
>>>> choose*
>>>> * *   cache_size to have "n modulo cache_size == 0": if this is*
>>>> * *   not the case, some elements will always stay in the pool and will*
>>>> * *   never be used.* The access to the per-lcore table is of course
>>>>  *   faster than the multi-producer/consumer pool. The cache can be
>>>>  *   disabled if the cache_size argument is set to 0; it can be useful to
>>>>  *   avoid losing objects in cache.
>>>> 
>>>> I wonder if someone can please explain the high-lightened sentence, how the
>>>> cache size affects the objects inside the ring.
>>> It has no effects upon the objects themselves. Having a cache is
>>> strongly recommended for performance reasons. Accessing a shared ring
>>> for a mempool is very slow compared to pulling packets from a per-core
>>> cache. To test this you can run testpmd using different --mbcache
>>> parameters.
>> Still, I didn't understand the sentence from above:
>> 
>> *It is advised to choose cache_size to have "n modulo cache_size == 0": if
>> this is* not the case, some elements will always stay in the pool and will*
>> never be used.*
>> 
> 
> This would be an artifact of the way in which the elements are taken
> from the pool ring. If a full cache-size burst of elements is not
> available in the ring, no elements from the ring are put in the cache.
> It just means that the pool can never fully be emptied. However, in most
> cases, even having the pool nearly empty indicates a problem, so
> practically I wouldn't be worried about this.
> 

But in case we tried to get cache size from pool and failed we will try to get the number on objects defined in rte_mempool_get_bulk, so in case rte_mempool_get() usage we will try to get one object out of the pool (ring) so also if there isn't cache size in the ring itself each core can get 1 object in each rte_memoool_get until the pool is empty, am I wrong?

>>> 
>>>> And also does it mean that
>>>> if I'm sharing pool between different cores can it be that a core sees the
>>>> pool as empty although it has objects in it?
>>>> 
>>> Yes, that can occur. You need to dimension the pool to take account of
>>> your cache usage.
>> 
>> can you please elaborate more on this issue? I'm working with multi-consumer
>> multi-producer pools, in my understanding object can or in lcore X cache or
>> in ring.
>> Each core when looking for objects in pool (ring) is looking at prod/cons
>> head/tail so how can it be that the cache of different cores affects this?
>> 
> 
> Mempool elements in the caches are free elements that are available for
> use. However, they are inaccessible to any core except the core which
> freed them to that cache. For a slightly simplified example, consider a
> pool with 256 elements, and a cache size of 64. If 4 cores all request 1
> buffer each, those four cores will each fill their caches and then take
> 1 element from those caches. This means that the ring will be empty even
> though only 4 buffers are actually in use - the other 63*4 buffers are
> in per-core caches. A 5th core which comes along and requests a buffer
> will be told that the pool is empty, despite there being 252 free
> elements.
> 
> Therefore you need to take account of the possibilities of buffers being
> stored in caches when dimensioning your mempool.
> 
> /Bruce