[dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec

[Yan Lei]

Hi Thomas,


I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3.


TL;DR


1. I was only able to generate 3K rules per second.

2. The maximum number of distinct rules the NIC can support seems to be 65536.


How can I increase the insertion rate? Any firmware/driver config I need to tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to generate 4 million distinct flows though...


Thanks in advance!



Initially, running


```

sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=1000000

```


failed after a few seconds and it gave


```
Flow can't be created 1 message: hardware refuses to create flow
EAL: Error - exiting with code: 1
  Cause: error in creating flow
```


Then I added a small debug patch (attached) and it showed that the error happens when creating the 65536th flow rule.


```
Flow can't be created 1 message: hardware refuses to create flow
EAL: Error - exiting with code: 1
  Cause: error in creating flow,flows generated: 65536
```


My guess is that the NIC can only accept 65536 concurrent rules. Once I changed the outer ip mask to 0xffff, the above command runs fine.


To see how many rules I can generate per second. I ran (with the outer ip mask 0xffff)


```

sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=65536

```


and it gives


```

:: Total flow insertion rate -> 3.015922 K/Sec
:: The time for creating 65536 in flows 21.730005 seconds
:: EAGAIN counter = 0
```
So 3 rules per sec. Which is close to what I observed before.

```
sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=100000
```
gives

```
:: Total flow insertion rate -> 0.949381 K/Sec
:: The time for creating 100000 in flows 105.331842 seconds
:: EAGAIN counter = 0
```
Have no idea why it's only 1k/sec in this case...

Thanks and cheers,
Lei


________________________________
From: users <users-bounces at dpdk.org> on behalf of Yan Lei <l.yan at epfl.ch>
Sent: Tuesday, April 14, 2020 1:20 PM
To: Thomas Monjalon
Cc: users at dpdk.org
Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec

Hi Thomas,

Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3).

Cheers,
Lei
________________________________
From: Thomas Monjalon <thomas at monjalon.net>
Sent: Tuesday, April 14, 2020 12:12:28 PM
To: Yan Lei
Cc: users at dpdk.org
Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec

Hi,

10/04/2020 20:11, Yan Lei:
> I am doing some study that requires inserting more than 1 million flow
> rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC.
>
> But I only managed to create around 3.3K rules per second.
> Below is the code I used to measure the insertion rate:

Please could you review this new application designed for such measure?
        https://patches.dpdk.org/patch/68058/

Any feedback about the above patch is welcome. Feel free to try and review it.