[dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec

[Tom Barbette]

Hi,

Le 19/04/2020 à 16:07, Wisam Monther a écrit :
> Hey Yan,
> 
> For FW steering there is HW limitation on number of flows that can be added on it.
> Which is table 0, and I think it's 65536
> 
> But to get Millions of rules use --group=1 which is SW steering.
What do you mean by SW steering?

Using group 1 we had better performance but only around 10K. I think the 
whole process lacks an update feature instead of delete+create, and the 
ability to batch rules.

> Moreover make sure you have enough memory in the app to have good insertion rate.
> 
> If you have enough 1G huge pages then it's ok.
> If you are working with 2M pages your command should be like this:
> 
> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000
> 
> BRs,
> Wisam Jaddo
> 
>> -----Original Message-----
>> From: Thomas Monjalon <thomas at monjalon.net>
>> Sent: Sunday, April 19, 2020 4:58 PM
>> To: Yan Lei <l.yan at epfl.ch>
>> Cc: users at dpdk.org; Wisam Monther <wisamm at mellanox.com>
>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K
>> per sec
>>
>> +Cc Wisam
>>
>> 16/04/2020 17:32, Yan Lei:
>>> Hi Thomas,
>>>
>>>
>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3.
>>>
>>>
>>> TL;DR
>>>
>>>
>>> 1. I was only able to generate 3K rules per second.
>>>
>>> 2. The maximum number of distinct rules the NIC can support seems to be
>> 65536.
>>>
>>>
>>> How can I increase the insertion rate? Any firmware/driver config I need to
>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to
>> generate 4 million distinct flows though...
>>>
>>>
>>> Thanks in advance!
>>>
>>>
>>>
>>> Initially, running
>>>
>>>
>>> ```
>>>
>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
>>> --ether --ipv4 --udp --queue --flows-count=1000000
>>>
>>> ```
>>>
>>>
>>> failed after a few seconds and it gave
>>>
>>>
>>> ```
>>> Flow can't be created 1 message: hardware refuses to create flow
>>> EAL: Error - exiting with code: 1
>>>    Cause: error in creating flow
>>> ```
>>>
>>>
>>> Then I added a small debug patch (attached) and it showed that the error
>> happens when creating the 65536th flow rule.

The first table is indeed limited to something around that number. But 
performance are already degrading before that point. Even with OFED 5 
and the firmware that comes with it.

>>>
>>>
>>> ```
>>> Flow can't be created 1 message: hardware refuses to create flow
>>> EAL: Error - exiting with code: 1
>>>    Cause: error in creating flow,flows generated: 65536 ```
>>>
>>>
>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I
>> changed the outer ip mask to 0xffff, the above command runs fine.
>>>
>>>
>>> To see how many rules I can generate per second. I ran (with the outer
>>> ip mask 0xffff)
>>>
>>>
>>> ```
>>>
>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
>>> --ether --ipv4 --udp --queue --flows-count=65536
>>>
>>> ```
>>>
>>>
>>> and it gives
>>>
>>>
>>> ```
>>>
>>> :: Total flow insertion rate -> 3.015922 K/Sec
>>> :: The time for creating 65536 in flows 21.730005 seconds
>>> :: EAGAIN counter = 0
>>> ```
>>> So 3 rules per sec. Which is close to what I observed before.
>>>
>>> ```
>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives
>>>
>>> ```
>>> :: Total flow insertion rate -> 0.949381 K/Sec
>>> :: The time for creating 100000 in flows 105.331842 seconds
>>> :: EAGAIN counter = 0
>>> ```
>>> Have no idea why it's only 1k/sec in this case...
>>>
>>> Thanks and cheers,
>>> Lei
>>>
>>>
>>> ________________________________
>>> From: users <users-bounces at dpdk.org> on behalf of Yan Lei
>>> <l.yan at epfl.ch>
>>> Sent: Tuesday, April 14, 2020 1:20 PM
>>> To: Thomas Monjalon
>>> Cc: users at dpdk.org
>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
>>> than 4K per sec
>>>
>>> Hi Thomas,
>>>
>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3).
>>>
>>> Cheers,
>>> Lei
>>> ________________________________
>>> From: Thomas Monjalon <thomas at monjalon.net>
>>> Sent: Tuesday, April 14, 2020 12:12:28 PM
>>> To: Yan Lei
>>> Cc: users at dpdk.org
>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
>>> than 4K per sec
>>>
>>> Hi,
>>>
>>> 10/04/2020 20:11, Yan Lei:
>>>> I am doing some study that requires inserting more than 1 million
>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5
>> NIC.
>>>>
>>>> But I only managed to create around 3.3K rules per second.
>>>> Below is the code I used to measure the insertion rate:
>>>
>>> Please could you review this new application designed for such measure?
>>>
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
>>>
>> hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m
>> ellanox.c
>>>
>> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149
>> 256f46
>>>
>> 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGyM
>> FaYgk%2BjSE
>>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0
>>>
>>> Any feedback about the above patch is welcome. Feel free to try and review
>> it.
>>
>>
>>
> 


Tom