[dpdk-dev] packet loss: multi-queue (RSS enabled)

[Jayakumar, Muthurajan]

Hi, 
Please find the attached paper http://kfall.net/ucbpage/papers/snc.pdf
Figures 4 and 5 refers about the degradation when the # of queues are increased. 
It refers sweet spot as 2 to 4 queues. 

Have you please verified with smaller # of queues please?

Thanks, 


-----Original Message-----
From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Hamid Ramazani
Sent: Tuesday, April 29, 2014 9:42 PM
To: Thomas Monjalon; dev at dpdk.org
Subject: [dpdk-dev] packet loss: multi-queue (RSS enabled)

Hi,
I tried a lot (more than a week) to solve the problem myself and not to bother the list, but I didn't succeed. Maybe other people have the same problem.

I have a simple program attached. It is intended for simple packet capturing; captures from interface and writes to memory, and frees the memory in the next loop.

I have a 10G 82599EB Intel SFI/SFP+ network interface for capturing packets.
As you may know, this network card supports up to 128 RSS queues.

This is just a test, so the packets being sent at 820Kpps (kilo packet per second). Each packet is 1500B (fixed size); it is 9.16 Gbit per second.
Of course when the packet per seconds goes up and packet size goes down (e.g. 400B per packet), it gets much worse.

When using one queue to receive, I receive all the packets, with no loss.
When I use more than one queue (e.g. 8 queues), with each thread running on a dedicated core, I have a considerable amount of loss.

Please note that:
1. The computer has 12 * 2.67GHz cores, and it does nothing else but capturing packets. The CPU is Intel Xeon X5650.
2. The operating system is Ubuntu 12.04.3 LTS

Attached file includes:
main.h
main.c
Makefile
./run.sh

It is configured to be run with 8 queues.

If you want to change the number of receive queues, please:
1. in main.c, change the value assigned to nb_rx_q_of_dev to the desired value.
2. change core mask in run.sh file (since there is SKIP_MASTER, you should give a core containing one more CPU than given number of queues).

I think there might be following problems:
1. the port configuration is not fine.
2. freeing memory has a considerable amount of overhead, and may be I shouldn't do that. But If I don't the pool will be full, won't be? Is there any other way?

Please help.
Thanks a lot in advance for your help and comments.

All the Best,
Hamid