[dpdk-dev] Polling too often at lower packet rates?

[Aaron Campbell]

Hi Stephen,

Thanks, that was an informative talk.  In this case, are you referring to your comments about the thermal budget?

That’s definitely interesting, but there must be more to it than that.  Again, if I loop over all 6 ports (i.e., continue to keep the CPU busy), it works around the problem.

I agree that adaptive polling makes sense and will look into it.  But will still take any further ideas on what is going on here.

-Aaron

> On Apr 8, 2015, at 3:00 PM, Stephen Hemminger <stephen at networkplumber.org> wrote:
> 
> We use adaptive polling loop similar to l3fwd-power example.
> See:
>   
> 
> http://video.fosdem.org/2015/devroom-network_management_and_sdn/ <http://video.fosdem.org/2015/devroom-network_management_and_sdn/>
> 
> On Wed, Apr 8, 2015 at 9:35 AM, Aaron Campbell <aaron at arbor.net <mailto:aaron at arbor.net>> wrote:
> Hi,
> 
> I have a machine with 6 DPDK ports (4 igb, 2 ixgbe), with 1.23Mpps traffic offered to only one of the 10G ports (the other 5 are unused).  I also have a program with a pretty standard looking DPDK receive loop, where it calls rte_eth_rx_burst() for each configured port.  If I configure the loop to read from all 6 ports, it can read the 1.23Mpps rate with no drops.  If I configure the loop to poll only 1 port (the ixgbe receiving the traffic), I lose about 1/3rd of the packets (i.e., the NIC drops ~400Kpps).
> 
> Another data point is that if I configure the loop to read from 3 out of the 6 ports, the drop rate is reduced to less than half (i.e., the NIC is only dropping ~190Kpps now).  So it seems that in this test, throughput improves by adding NICs, not removing them, which is counter-intuitive.  Again, I get no drops when polling all 6 ports.  Note, the burst size is 32.
> 
> I did find a reference to a similar issue in a recent paper (http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ICN2015.pdf <http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ICN2015.pdf>), Section III, which states:
> 
> "The DPDK L2FWD application initially only managed to forward 13.8 Mpps in the single direction test at the maximum CPU frequency, a similar result can be found in [11]. Reducing the CPU frequency increased the throughput to the expected value of 14.88 Mpps. Our investigation of this anomaly revealed that the lack of any processing combined with the fast CPU caused DPDK to poll the NIC too often. DPDK does not use interrupts, it utilizes a busy wait loop that polls the NIC until at least one packet is returned. This resulted in a high poll rate which affected the throughput. We limited the poll rate to 500,000 poll operations per second (i.e., a batch size of about 30 packets) and achieved line rate in the unidirectional test with all frequencies. This effect was only observed with the X520 NIC, tests with X540 NICs did not show this anomaly.”
> 
> Another reference, from this mailing list last year (http://wiki.dpdk.org/ml/archives/dev/2014-January/001169.html <http://wiki.dpdk.org/ml/archives/dev/2014-January/001169.html>):
> 
> "I suggest you to check average burst sizes on receive queues. Looks like I stumbled upon a similar issue several times. If you are calling rte_eth_rx_burst too frequently, NIC begins losing packets no matter how many CPU horse power you have (more you have, more it loses, actually). In my case this situation occured when average burst size is less than 20 packets or so. I'm not sure what's the reason for this behavior, but I observed it on several applications on Intel 82599 10Gb cards.”
> 
> So I’m wondering if anyone can explain at a lower level what happens when you poll “too often”, and if there are any practical workarounds.  We’re using this same program and DPDK version to process 10G line-rate in other scenarios, so I’m confident that the overall packet capture architecture is sound.
> 
> -Aaron
>