[dpdk-users] Poor OVS-DPDK performance

[Mooney, Sean K]

Some comments inline

From: abhishek jain [mailto:ashujain9727 at gmail.com]
Sent: Monday, December 11, 2017 6:55 AM
To: Mooney, Sean K <sean.k.mooney at intel.com>
Cc: users at dpdk.org
Subject: Re: Poor OVS-DPDK performance

Hi Sean
Thanks for looking into it.
You got it right, I'm targeting phy-VM-phy numbers
I'm having 8 core host with HT turned on with 16 threads.Below is the output describing it details.

[Mooney, Sean K] given the resource limits of the plathform in your case I would recommend staring with less core for the vswitch
to leave as many free for vms as possible so I would set
{dpdk-init="true", dpdk-lcore-mask="0x1", dpdk-socket-mem="1024", pmd-cpu-mask="0x202"}
And set isolcpus to isolcpus=1,9
if you want to further incress the performace you can also add the following to your kernel cmdlin
nohz=on nohz_full="1-15" rcu_nocbs="1-15" nosoftlockup mce=ignore_ce audit=0 elevator=deadline
if you are using openstack you nova vcpu_pin_set should be vcpu_pin_set=”2-7,10-15”
so core 0 and 8(it hyper thread) are used for the os core 1 and 9 are used for dpdk pmds leaving 6 physical cores/12 hw threads free for vms.
nohz is disableing kernel things on core that are idel(is is a power saving feature more or less) and nohz_full disable ticks on cores with 1 active process
and no other process pending scheduling to the same core. This is an important optimization for realtime systems but it does not hurt to do for dpdk either.

 lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 86
Stepping:              3
CPU MHz:               800.000
BogoMIPS:              4199.99
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              12288K
NUMA node0 CPU(s):     0-15
Please let me know the appropriate configuration for better performance with this available setup.Also find the attached notepad file covering all the OVS-DPDK details of your queries in the above mail.

One more query,I have configured my phy interface with vfio DPDK compatible driver,below is the output ..

dpdk_nic_bind --status

Network devices using DPDK-compatible driver
============================================
0000:03:00.0 'Ethernet Connection X552/X557-AT 10GBASE-T' drv=vfio-pci unused=
0000:03:00.1 'Ethernet Connection X552/X557-AT 10GBASE-T' drv=vfio-pci unused=
0000:05:00.1 'I350 Gigabit Network Connection' drv=vfio-pci unused=
Once I have configured the vfio,I'm not able to receive any packets on that particular physical interface.
[Mooney, Sean K] this is expected. The vfio-pci driver is not a network adapter driver it is a generic pci device driver
to facilitate control of the device form userspace. As a result it does not provide netdevs for physical functions
or attach those interfaces to the kernel networking stack. With ovs 2.4 these interfaces are added to bridges
using the dpdkX naming scheme.  The dpdkX naming scheme is rather annoying to explain but basically the X is the index into the
Array of device bound to a dpdk driver sorted by pci address and filtered by the optional dpdk whitelist.
In the above case dpdk0 = 0000:03:00.0, dpdk1 = 0000:03:00.1, dpdk2 = 0000:05:00.1
If –w 0000:05:00.1 was added to the dpdk section of the ovs-dpdk commandlin then that would whitelist only
0000:05:00.1 and dpdk0 would map to 0000:05:00.1
Reviewing the attached files I noticed that while all the bridge are correctly configured to use the netdev
datapath and are correctly connected via patch ports the br-prv does not have a physical interface attached.
it instand has an ovs internal port"p_eeee51a2-0" which from the name scheme I assume means you deployed with fule.
if that assumption is correct I belive the p_eeee51a2-0 will be added to a linux bridge and that bridce connect to a physical interfaces.
intenal interfaces are not dpdk accelerated when using ovs-dpdk and fallback to the legacy netdev implementation which runs in the vswitchd main thread.
the port stats seam to also show that this interface is actively sending and reciving packets
{collisions=0, rx_bytes=1780832, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=41812, tx_bytes=184800, tx_dropped=0, tx_errors=0, tx_packets=3769}
status              : {driver_name=tun, driver_version="1.6", firmware_version=""}

I would suggest removing the interface
sudo ovs-vsctl rm-port p_eeee51a2-0
and replace it with a dpdk interface
sudo ovs-vsctl add-port br-prv dpdk0 -- set interface dpdk0 type=dpdk
once this is done you likely need to restart the vswitchd for it ot take effect as ovs 2.4 does not
support adding dpdk physical interfaces are the vswith is running unless they were bound to the
vfio-pci driver then the vswitch was started.


Thanks again for you time.

Regards
Abhishek Jain






On Fri, Dec 8, 2017 at 6:11 PM, Mooney, Sean K <sean.k.mooney at intel.com<mailto:sean.k.mooney at intel.com>> wrote:
Hi can you provide the qemu commandline you are using for the vm
As well as the following commands

Ovs-vsctl show
Ovs-vsctl list bridge
Ovs-vsctl list interface
Ovs-vsctl list port

Basically with the topology info above I want to confirm:

-          All bridges are interconnected by patch ports not veth,

-          All bridges are datapath type netdev

-          Vm is connected by vhost-user interfaces not kernel vhost(it still works with ovs-dpdk but is really slow)

Looking at you core masks the vswitch I incorrectly tunned

{dpdk-init="true", dpdk-lcore-mask="7fe", dpdk-socket-mem="1024", pmd-cpu-mask="1800"}
root at node-3:~#

You have configured 10 lcores which are not used for packet processing and only the 1st of which will actually  be used by ovs-dpdk
And you have configured a single core for the pmd

You have not said if you are on a multi socket system of single but assuming that it is a single socket system try this instead.

{dpdk-init="true", dpdk-lcore-mask="0x2", dpdk-socket-mem="1024", pmd-cpu-mask="0xC"}
Above im assuming core 0 is used by os  core 1 will be used for lcore thread and cores 2 and 3 will be used for pmd threads which does all the packet forwarding.

If you have hyperthreads turn on on your host add the hyperthread siblings to the pmd-cpu-mask
For example if you had a 16 core cpu with 32 threads the pmd-cpu-mask should be “0x60006”

On the kernel cmdline change the isolcpus to isolcpus=2,3 or isolcpus=2,3,18,19 for hyper threading.

With the HT config you should be able to handel upto 30mpps on a 2.2GHz cpu assuming you compiled ovs and dpdk with “-fPIC -02 –march=native” and link statically.
If you have a faster cpu you should get more but as a rough estimate when using ovs 2.4 and dpdk 2.0 you should
expect between 6.5-8mpps phy to phy per physical core + an additional 70-80%  if hyper thereading is used.

Your phy vm phy numbers will be a little lower as the vhost-user pmd takes more clock cycles to process packets then
The physical nic drivers do but that should help set your expectations.

Ovs 2.4 and dpdk 2.0 are quite old at this point but they still should give a significant performance increase over kernel ovs.




From: abhishek jain [mailto:ashujain9727 at gmail.com<mailto:ashujain9727 at gmail.com>]
Sent: Friday, December 8, 2017 9:34 AM
To: users at dpdk.org<mailto:users at dpdk.org>; Mooney, Sean K <sean.k.mooney at intel.com<mailto:sean.k.mooney at intel.com>>
Subject: Re: Poor OVS-DPDK performance

Hi Team
Below is my OVS configuration..

root at node-3:~# ovs-vsctl get Open_vSwitch . other_config
{dpdk-init="true", dpdk-lcore-mask="7fe", dpdk-socket-mem="1024", pmd-cpu-mask="1800"}
root at node-3:~#

root at node-3:~#
root at node-3:~# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.13.0-137-generic root=/dev/mapper/os-root ro console=tty0 net.ifnames=0 biosdevname=0 rootdelay=90 nomodeset root=UUID=2949f531-bedc-47a0-a2f2-6ebf8e1d1edb iommu=pt intel_iommu=on isolcpus=11,12,13,14,15,16

root at node-3:~#
root at node-3:~# ovs-appctl dpif-netdev/pmd-stats-show
main thread:
        emc hits:2
        megaflow hits:0
        miss:2
        lost:0
        polling cycles:99607459 (98.80%)
        processing cycles:1207437 (1.20%)
        avg cycles per packet: 25203724.00 (100814896/4)
        avg processing cycles per packet: 301859.25 (1207437/4)
pmd thread numa_id 0 core_id 11:
        emc hits:0
        megaflow hits:0
        miss:0
        lost:0
        polling cycles:272926895316 (100.00%)
        processing cycles:0 (0.00%)
pmd thread numa_id 0 core_id 12:
        emc hits:0
        megaflow hits:0
        miss:0
        lost:0
        polling cycles:240339950037 (100.00%)
        processing cycles:0 (0.00%)
root at node-3:~#

root at node-3:~#
root at node-3:~# grep -r Huge /proc/meminfo
AnonHugePages:     59392 kB
HugePages_Total:    5126
HugePages_Free:     4870
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
root at node-3:~#
I'm using OVS version 2.4.1
Regards
Abhishek Jain

On Fri, Dec 8, 2017 at 11:18 AM, abhishek jain <ashujain9727 at gmail.com<mailto:ashujain9727 at gmail.com>> wrote:
Hi Team
Currently I have OVS-DPDK setup configured on Ubuntu 14.04.5 LTS. I'm also having one vnf with vhost interfaces mapped to OVS bridge br-int.
However when I'm performing throughput with the same vnf,I'm getting very less throughput.

Please provide me some pointers to boost the performance of vnf with OVS-DPDK configuration.
Regards
Abhishek Jain