[dpdk-users] DPDK TX problems

[Hrvoje Habjanic]

Hi.

I did write an application using dpdk 17.11 (did try also with 18.11),
and when doing some performance testing, i'm seeing very odd behavior.
To verify that this is not because of my app, i did the same test with
l2fwd example app, and i'm still confused by results.

In short, i'm trying to push a lot of L2 packets through dpdk engine -
packet processing is minimal. When testing, i'm starting with small
number of packets-per-second, and then gradually increase it to see
where is the limit. At some point, i do reach this limit - packets start
to get dropped. And this is when stuff become weird.

When i reach peek packet rate (at which packets start to get dropped), i
would expect that reducing packet rate will remove packet drops. But,
this is not the case. For example, let's assume that peek packet rate is
3.5Mpps. At this point everything works ok. Increasing pps to 4.0Mpps,
makes a lot of dropped packets. When reducing pps back to 3.5Mpps, app
is still broken - packets are still dropped.

At this point, i need to drastically reduce pps (1.4Mpps) to make
dropped packets go away. Also, app is unable to successfully forward
anything beyond this 1.4M, despite the fact that in the beginning it did
forward 3.5M! Only way to recover is to restart the app.

Also, sometimes, the app just stops forwarding any packets - packets are
received (as seen by counters), but app is unable to send anything back.

As i did mention, i'm seeing the same behavior with l2fwd example app. I
did test dpdk 17.11 and also dpdk 18.11 - the results are the same.

My test environment is HP DL380G8, with 82599ES 10Gig (ixgbe) cards,
connected with Cisco nexus 9300 sw. On the other side is ixia test
appliance. Application is run in virtual machine (VM), using KVM
(openstack, with sriov enabled, and numa restrictions). I did check that
VM is using only cpu's from NUMA node on which network card is
connected, so there is no cross-numa traffic. Openstack is Queens,
Ubuntu is Bionic release. Virtual machine is also using ubuntu bionic as OS.

I do not know how to debug this? Does someone else have the same
observations?

Regards,

H.