[dpdk-users] multiple use rte_pktmbuf_free occurs competition

forsakening at sina.cn forsakening at sina.cn
Sat Aug 20 10:19:07 CEST 2016

Hi :
I use the below struct to handle the internet packet based on the dpdk-16.04 interfaces. 
And my hardware: Intel(R) Xeon(R) CPU E5-2618L v3 @ 2.30GHz , which has 32 logic cores .
And my platform: CentOs 7.0.

[stuct removed cuz not allowed pic.]

Internet Packet --> NIC --> DPDK-RX --> Dispatch --> Handle Packet Threads (1 to N) --> rte_pktmbuf_free

And now I use about 12 or more handle threads, and these threads are set affinity to my intel logic core, also, in order not to schedule by other thread, i isolate these logic cores out.

Cuz the Packet may not be very standard Ethernet Packet, or My 82599ES NIC cannot RSS the packet well, I use only one Mempool, and setup only one rx que.
To handle these packet, I use one dispatch thread to "Decode" the packet and  dispatch them to my handle threads.
And then , question comes,when these handle threads free the packet, they are “blocked” or "slow" on rte_pktmbuf_free interface.

Below is my "perf result" of one thread:
Samples: 488K of event 'cycles', Event count (approx.): 22076989633                                                                                                                                                                                                                                                        
 78.92%  libvst_driver.so     [.] rte_pktmbuf_free
 .....Other Info

you can see the free interface slow my thread.....

and, annonate of the rte_pktmbuf_free like this:
 * If there are other enqueues in progress that preceded us,
 * we need to wait for them to complete
while (unlikely(r->prod.tail != prod_head)) {

/* Set RTE_RING_PAUSE_REP_COUNT to avoid spin too long waiting
 * for other thread finish. It gives pre-empted thread a chance
 * to proceed and finish with ring dequeue operation. */
    ++rep == RTE_RING_PAUSE_REP_COUNT) {
rep = 0;

According to the hotspot,the free interface is waiting for other thread to compelete the enque process. 

I know the thread of my  free thread are little more , but can anybody give me some advise about how can i  use the free interface more effectively  ?

Thanks very much !!

Best Regards
forsakening at sina.cn 
MYTtech, NanJing, CHN

More information about the users mailing list