[dpdk-dev] Packets reorder in the second pass on mempool

[Masoud Moshref Javadi]

Hi
I'm new to dpdk and I hope you can solve my weird problem.

I see packet reordering correlated with the batchsize and mempool size.

I have a very simple setting of a sender and a receiver connected with a 
simple 10G switch.
The sender sends udp packets at 14.88MPPS and just puts a 32bit sequence 
number in the udp packets.
The receiver reads packets and expects the packets to be in order.
However, the receiver sees packets in a reordered way like this (note 
that it starts from 7808 in different runs):

expected, seen , seen-expected
7808, 7936 = 128
7978, 8106 = 128
8145, 8273 = 128
8308, 8436 = 128
8448, 8320 = -128
8391, 8519 = 128
8576, 8448 = -128
8518, 8646 = 128
8704, 8576 = -128
....

The batchsize at sender is 128.
The configuration of ports and mempools are similar to basicfwd.c in 
basic forwarding example of dpdk.
Interestingly if I change the size of mempool, the beginning of packet 
reordering changes from 7808.

#define NUM_MBUFS 8192
#define MBUF_SIZE (1600 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
#define MBUF_CACHE_SIZE 250
#define BURST_SIZE 128

mbuf_pool = rte_mempool_create("MBUF_POOL",
                                        NUM_MBUFS * nb_ports,
                                        MBUF_SIZE,
                                        MBUF_CACHE_SIZE,
                                        sizeof(struct 
rte_pktmbuf_pool_private),
                                        rte_pktmbuf_pool_init, NULL,
                                        rte_pktmbuf_init,      NULL,
                                        1,
                                        0);

The main sender loop is as follows:

  for (;pkts_num<target;) {
                         /* Get burst of RX packets, from first port of 
pair. */
                         rte_mempool_sc_get_bulk(mbuf_pool, 
(void**)&bufs, BURST_SIZE);
                         /* Send burst of TX packets, to second port of 
pair. */
                         for (i=0; i< BURST_SIZE; i++){
                                 m = bufs[i];
                                 m->data_len = 60;
                                 m->pkt_len  = 60;
                                 rte_memcpy((uint8_t *)m->buf_addr + 
m->data_off,bufpkt, 60);
                                 uint32_t * data = (uint32_t 
*)(rte_ctrlmbuf_data(m) + data_offset);
                                 *data = pkts_num+i;
                         }

                         uint16_t nb_tx = 0;
                         while (nb_tx < BURST_SIZE){
                                 nb_tx+= rte_eth_tx_burst(port, 0, bufs 
+ nb_tx, BURST_SIZE - nb_tx);
                         }
                         pkts_num += nb_tx;
}

Regards