rte_eth_tx_burst() always returns 0 in tight loop

[Antonio Di Bacco]

I have an E810-C card with  iavf-4.4.2.1  ice-1.8.8 drivers.

My tx_free_threshold is 8 together with tx_rs_thresh.

I have a tight loop sending BURSTs of 8 packets, each packet is 9014
bytes long (8 packets take 6 usecs to be serialized).
If I put the rte_delay_us_block to 7 then everything works fine, every
cycle 8 packets are transmitted.
If I lower  the rte_delay_us_block to 1 usec, then I observe that the
first FOR cycle is ok, nb_tx is 8 as expected, then, the second cycle
prints a 7 while all subsequent cycles print Z (zero packets sent). I
know that 1 usec delay is too small and I expect that no packets are
transmitted for some cycles but I don't understand why I get an nb_tx
set to 0 forever after the first two cycles.

for (;;)
{
        rte_spinlock_lock(&spinlock_conf[src_port]) ;
        const uint16_t nb_tx = rte_eth_tx_burst(src_port, 0, tx_bufs,
BURST_SIZE);
        rte_spinlock_unlock(&spinlock_conf[src_port]);

        rte_delay_us_block(7);  // tested with 1

        if (nb_tx == 0)
            printf("Z");
        else if (nb_tx < BURST_SIZE)
            printf("nb_tx %d\n", nb_tx);

        tx += nb_tx;

        if (unlikely(nb_tx < BURST_SIZE)) {
            uint16_t buf;

            for (buf = nb_tx; buf < BURST_SIZE; buf++)
                rte_pktmbuf_free(tx_bufs[buf]);
        }
}

On Wed, Jul 6, 2022 at 5:00 PM Stephen Hemminger
<stephen at networkplumber.org> wrote:
>
> On Wed, 6 Jul 2022 09:21:28 +0200
> Antonio Di Bacco <a.dibacco.ks at gmail.com> wrote:
>
> > I wonder why calling eth_dev_tx_burst in a tight loop doesn't allow to
> > write the packets into the transmit buffer. Only solution I found is
> > to include a small delay after the tx_burst that is less than the
> > estimated serialization time of the packet in order to be able to
> > saturate the ethernet line.
> >
> > Anyway I wonder if this is the right approach.
> >
> > Thx,
> > Antonio.
> >
> > On Sun, Jul 3, 2022 at 10:19 PM Gábor LENCSE <lencse at hit.bme.hu> wrote:
> > >
> > > Dear Antonio,
> > >
> > > According to my experience, the rte_eth_tx_burst() function reports the
> > > packets as "sent" (by a non-zero return value), when they are still in
> > > the transmit buffer.
> > >
> > > (If you are interested in the details, you can see them in Section 3.6.5
> > > of this paper: http://www.hit.bme.hu/~lencse/publications/e104-b_2_128.pdf )
> > >
> > > Therefore, I think that the return value of 0 may mean that
> > > rte_eth_tx_burst() can't even commit itself for the future delivery of
> > > the packets. I could only guess why. E.g. all its resources have been
> > > exhausted.
> > >
> > > Best regards,
> > >
> > > Gábor
> > >
> > >
> > > 7/3/2022 5:57 PM keltezéssel, Antonio Di Bacco írta:
> > > > I'm trying to send packets continuously in a  tight loop with a burst
> > > > size of 8 and packets are 9600 bytes long.
> > > > If I don't insert a delay after the rte_eth_tx_burst it always returns 0.
> > > >
> > > > What's the explanation of this behaviour ?
> > > >
> > > > Best regards,
> > > > Antonio.
> > >
>
> Which driver? How did you set the tx_free threshold.
> The driver will need to cleanup already transmitted packets.