[dpdk-users] Low Rx throughput when using Mellanox ConnectX-3 card with DPDK

[Shihabur Rahman Chowdhury]

Hi Dave,

We've checked this and we are using the lcores on the NUMA node where the
PCI interface to the NIC is.

Shihabur Rahman Chowdhury
David R. Cheriton School of Computer Science
University of Waterloo



On Wed, Apr 12, 2017 at 9:56 PM, Dave Wallace <dwallacelf at gmail.com> wrote:

> I have encountered a similar issue in the past on a system configuration
> where the PCI interface to the NIC was on the other NUMA node.
>
> Something else to check...
> -daw-
>
>
>
> On 04/12/2017 08:06 PM, Shihabur Rahman Chowdhury wrote:
>
>> We've disabled the pause frames. That also disables flow control I assume.
>> Correct me if I am wrong.
>>
>> On the pktgen side, the dropped field and overrun field for Rx keeps
>> increasing for ifconfig. Btw, both overrun and dropped fields have the
>> exact same value always.
>>
>> Our ConnectX3 NIC has just a single 10G port. So there is no way to create
>> such loopback connection.
>>
>> Thanks
>>
>> Shihabur Rahman Chowdhury
>> David R. Cheriton School of Computer Science
>> University of Waterloo
>>
>>
>>
>> On Wed, Apr 12, 2017 at 6:41 PM, Wiles, Keith <keith.wiles at intel.com>
>> wrote:
>>
>> On Apr 12, 2017, at 4:00 PM, Shihabur Rahman Chowdhury <
>>>>
>>> shihab.buet at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> We are running a simple DPDK application and observing quite low
>>>> throughput. We are currently testing a DPDK application with the
>>>>
>>> following
>>>
>>>> setup
>>>>
>>>> - 2 machines with 2xIntel Xeon E5-2620 CPUs
>>>> - Each machine with a Mellanox single port 10G ConnectX3 card
>>>> - Mellanox DPDK version 16.11
>>>> - Mellanox OFED 4.0-2.0.0.1 and latest firmware for ConnectX3
>>>>
>>>> The application is doing almost nothing. It is reading a batch of 64
>>>> packets from a single rxq, swapping the mac of each packet and writing
>>>> it
>>>> back to a single txq. The rx and tx is being handled by separate lcores
>>>>
>>> on
>>>
>>>> the same NUMA socket. We are running pktgen on another machine. With 64B
>>>> sized packets we are seeing ~14.8Mpps Tx rate and ~7.3Mpps Rx rate in
>>>> pktgen. We checked the NIC on the machine running the DPDK application
>>>> (with ifconfig) and it looks like there is a large number of packets
>>>>
>>> being
>>>
>>>> dropped by the interface. Our connectx3 card should be theoretically be
>>>> able to handle 10Gbps Rx + 10Gbps Tx throughput (with channel width 4,
>>>>
>>> the
>>>
>>>> theoretical max on PCIe 3.0 should be ~31.2Gbps). Interestingly, when Tx
>>>> rate is reduced in pktgent (to ~9Mpps), the Rx rate increases to ~9Mpps.
>>>>
>>> Not sure what is going on here, when you drop the rate to 9Mpps I assume
>>> you stop getting missed frames.
>>> Do you have flow control enabled?
>>>
>>> On the pktgen side are you seeing missed RX packets?
>>> Did you loopback the cable from pktgen machine to the other port on the
>>> pktgen machine and did you get the same Rx/Tx performance in that
>>> configuration?
>>>
>>> We would highly appriciate if we could get some pointers as to what can
>>>>
>>> be
>>>
>>>> possibly causing this mismatch in Rx and Tx. Ideally, we should be able
>>>>
>>> to
>>>
>>>> see ~14Mpps Rx well. Is it because we are using a single port? Or
>>>>
>>> something
>>>
>>>> else?
>>>>
>>>> FYI, we also ran the sample l2fwd application and test-pmd and got
>>>> comparable results in the same setup.
>>>>
>>>> Thanks
>>>> Shihab
>>>>
>>> Regards,
>>> Keith
>>>
>>>
>>>
>