[dpdk-users] Iv40evf performance degradation when upgrade DPDK to 19-11

[Mohamed Mohamed]

Hi All:
 we are in the process of upgrading DPDK to 19-11 while testing i40evf driver with multiple queues we are seeing performance drop significantly the driver log seems mostly clean 10-21-2020  15:53:26.474906 i40evf_handle_aq_msg(): Failed to read msg from AdminQ,ret: -5710-21-2020  15:53:26.635553 i40evf_check_api_version(): Peer is Linux PF host10-21-2020  15:53:26.645551 EAL: PCI device 0000:00:0d.0 on NUMA socket -110-21-2020  15:53:26.645554 EAL:   Invalid NUMA socket, default to 010-21-2020  15:53:26.645556 EAL:   probe driver: 8086:154c net_i40e_vf10-21-2020  15:53:26.695593 i40evf_handle_aq_msg(): Failed to read msg from AdminQ,ret: -5710-21-2020  15:53:26.858821 i40evf_check_api_version(): Peer is Linux PF host10-21-2020  15:53:26.868819 EAL: PCI device 0000:00:0e.0 on NUMA socket -110-21-2020  15:53:26.868821 EAL:   Invalid NUMA socket, default to 010-21-2020  15:53:26.868823 EAL:   probe driver: 8086:154c net_i40e_vf10-21-2020  15:53:26.918849 i40evf_handle_aq_msg(): Failed to read msg from AdminQ,ret: -5710-21-2020  15:53:27.81824 i40evf_check_api_version(): Peer is Linux PF host10-21-2020  15:53:27.124821 set_rxtx_funcs(): virtio: using standard Tx path on port 010-21-2020  15:53:27.124853 set_rxtx_funcs(): virtio: using mergeable buffer Rx path on port 010-21-2020  15:53:27.125810 set_rxtx_funcs(): virtio: using standard Tx path on port 110-21-2020  15:53:27.125822 set_rxtx_funcs(): virtio: using mergeable buffer Rx path on port 110-21-2020  15:53:27.126606 set_rxtx_funcs(): virtio: using standard Tx path on port 210-21-2020  15:53:27.126615 set_rxtx_funcs(): virtio: using mergeable buffer Rx path on port 210-21-2020  15:53:27.141849 i40evf_handle_aq_msg(): Failed to read msg from AdminQ,ret: -57
from DPDK stats I see                rx_dropped_packets : 7423883
               rx_missed_errors : 7505403
               
in one run I have seen more errors in the log but that happened for me once not consistent 10-20-2020  13:49:59.217581 i40evf_stop_queues(): Fail to stop queue 010-20-2020  13:49:59.217582 i40evf_execute_vf_cmd(): fail to send cmd 910-20-2020  13:49:59.217584 i40evf_switch_queue(): fail to switch RX 1 off10-20-2020  13:49:59.217585 i40evf_dev_rx_queue_stop(): Failed to switch RX queue 1 off10-20-2020  13:49:59.217587 i40evf_stop_queues(): Fail to stop queue 110-20-2020  13:49:59.217588 i40evf_execute_vf_cmd(): fail to send cmd 910-20-2020  13:49:59.217590 i40evf_switch_queue(): fail to switch RX 2 off10-20-2020  13:49:59.217591 i40evf_dev_rx_queue_stop(): Failed to switch RX queue 2 off

I can't build any theory based on the above one time instance
Can you pls provide some guidance how we can debug this further and how ?
Thanks,Mohamed