<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="en-IL" link="#0563C1" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> My question was theorical, is it possible to do the same load balance with the rss flow for vf as for pf for inner ip over gre, but then I tried and failed to initiate the testpmd application and also my own application.<o:p></o:p> <o:p> </o:p> I sent a separate email about it, but still didn’t succeed to find the problem – with mail title “sriov net_mlx5 error - Unexpected error in DR drop action support detection”<o:p></o:p> That was the content of the mail -<o:p></o:p> <o:p> </o:p> <ol style="margin-top:0cm" start="1" type="1"> <li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Dpdk version: 19.11 (dpdk-stable-19.11.11)<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Version of rdma core is 26.0 (from config.h #define PACKAGE_VERSION "26.0")<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">“Unexpected error in DR drop action support detection” message is coming from rte_eal_init<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">“DPDK: Error starting or stopping port -12” is coming from rte_eth_dev_start <o:p></o:p></li></ol> <o:p> </o:p> Here is the code of the program I run ( I run it with one rx queue and one tx queue), same program run for real interface (pf not vf) <o:p></o:p> I run it on vm (over esxi – vmware ) I don’t have access to host<o:p></o:p> <a href="https://coliru.stacked-crooked.com/a/6e2b9cd6cf047048">https://coliru.stacked-crooked.com/a/6e2b9cd6cf047048</a><o:p></o:p> Here is the output of the program I run<o:p></o:p> /home/deployment/testReceiver --log-level='.*',8 -l 7,0-15 -w 0b:00.0<o:p></o:p> <o:p> </o:p> Testpmd also failed to initiate with same parmeters<o:p></o:p> <o:p> </o:p> (I succeeded to run on another host with pf with dpdk19, and also +/- same code with dpdk17 with vf and pf.)<o:p></o:p> <o:p> </o:p> Some information about the system:<o:p></o:p> Network devices using kernel driver<o:p></o:p> ===================================<o:p></o:p> 0000:0b:00.0 'MT27800 Family [ConnectX-5 Virtual Function] 1018' if=ens192 drv=mlx5_core unused=igb_uio<o:p></o:p> 0000:13:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens224 drv=vmxnet3 unused=igb_uio *Active*<o:p></o:p> 0000:1b:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens256 drv=vmxnet3 unused=igb_uio *Active*<o:p></o:p> <o:p> </o:p> ethtool -i ens192<o:p></o:p> driver: mlx5_core<o:p></o:p> version: 5.0-0<o:p></o:p> firmware-version: 16.29.1016 (HPE0000000009)<o:p></o:p> expansion-rom-version:<o:p></o:p> bus-info: 0000:0b:00.0<o:p></o:p> supports-statistics: yes<o:p></o:p> supports-test: yes<o:p></o:p> supports-eeprom-access: no<o:p></o:p> supports-register-dump: no<o:p></o:p> supports-priv-flags: yes<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Some output logs:<o:p></o:p> <o:p> </o:p> EAL: Detected lcore 0 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 1 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 2 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 3 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 4 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 5 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 6 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 7 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 8 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 9 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 10 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 11 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 12 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 13 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 14 as core 0 on socket 0<o:p></o:p> EAL: Detected lcore 15 as core 0 on socket 0<o:p></o:p> EAL: Support maximum 128 logical core(s) by configuration.<o:p></o:p> EAL: Detected 16 lcore(s)<o:p></o:p> EAL: Detected 1 NUMA nodes<o:p></o:p> EAL: Ask a virtual area of 0x5000 bytes<o:p></o:p> EAL: Virtual area found at 0x100000000 (size = 0x5000)<o:p></o:p> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket<o:p></o:p> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)<o:p></o:p> EAL: VFIO PCI modules not loaded<o:p></o:p> EAL: Bus pci wants IOVA as 'DC'<o:p></o:p> EAL: Buses did not request a specific IOVA mode.<o:p></o:p> EAL: IOMMU is not available, selecting IOVA as PA mode.<o:p></o:p> EAL: Selected IOVA mode 'PA'<o:p></o:p> EAL: Probing VFIO support...<o:p></o:p> EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)<o:p></o:p> EAL: VFIO modules not loaded, skipping VFIO support...<o:p></o:p> EAL: Ask a virtual area of 0x2e000 bytes<o:p></o:p> EAL: Virtual area found at 0x100005000 (size = 0x2e000)<o:p></o:p> EAL: Setting up physically contiguous memory...<o:p></o:p> EAL: Setting maximum number of open files to 4096<o:p></o:p> EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824<o:p></o:p> EAL: Creating 4 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824<o:p></o:p> EAL: Ask a virtual area of 0x1000 bytes<o:p></o:p> EAL: Virtual area found at 0x100033000 (size = 0x1000)<o:p></o:p> EAL: Memseg list allocated: 0x100000kB at socket 0<o:p></o:p> EAL: Ask a virtual area of 0x800000000 bytes<o:p></o:p> EAL: Virtual area found at 0x140000000 (size = 0x800000000)<o:p></o:p> EAL: Ask a virtual area of 0x1000 bytes<o:p></o:p> EAL: Virtual area found at 0x940000000 (size = 0x1000)<o:p></o:p> EAL: Memseg list allocated: 0x100000kB at socket 0<o:p></o:p> EAL: Ask a virtual area of 0x800000000 bytes<o:p></o:p> EAL: Virtual area found at 0x980000000 (size = 0x800000000)<o:p></o:p> EAL: Ask a virtual area of 0x1000 bytes<o:p></o:p> EAL: Virtual area found at 0x1180000000 (size = 0x1000)<o:p></o:p> EAL: Memseg list allocated: 0x100000kB at socket 0<o:p></o:p> EAL: Ask a virtual area of 0x800000000 bytes<o:p></o:p> EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)<o:p></o:p> EAL: Ask a virtual area of 0x1000 bytes<o:p></o:p> EAL: Virtual area found at 0x19c0000000 (size = 0x1000)<o:p></o:p> EAL: Memseg list allocated: 0x100000kB at socket 0<o:p></o:p> EAL: Ask a virtual area of 0x800000000 bytes<o:p></o:p> EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)<o:p></o:p> EAL: TSC frequency is ~2000000 KHz<o:p></o:p> EAL: Master lcore 0 is ready (tid=7ffff7fe7900;cpuset=[0])<o:p></o:p> EAL: lcore 1 is ready (tid=7ffff4c81700;cpuset=[1])<o:p></o:p> EAL: lcore 11 is ready (tid=7fffefc77700;cpuset=[11])<o:p></o:p> EAL: lcore 10 is ready (tid=7ffff0478700;cpuset=[10])<o:p></o:p> EAL: lcore 14 is ready (tid=7fffee474700;cpuset=[14])<o:p></o:p> EAL: lcore 2 is ready (tid=7ffff4480700;cpuset=[2])<o:p></o:p> EAL: lcore 3 is ready (tid=7ffff3c7f700;cpuset=[3])<o:p></o:p> EAL: lcore 4 is ready (tid=7ffff347e700;cpuset=[4])<o:p></o:p> EAL: lcore 15 is ready (tid=7fffedc73700;cpuset=[15])<o:p></o:p> EAL: lcore 12 is ready (tid=7fffef476700;cpuset=[12])<o:p></o:p> EAL: lcore 9 is ready (tid=7ffff0c79700;cpuset=[9])<o:p></o:p> EAL: lcore 7 is ready (tid=7ffff1c7b700;cpuset=[7])<o:p></o:p> EAL: lcore 8 is ready (tid=7ffff147a700;cpuset=[8])<o:p></o:p> EAL: lcore 5 is ready (tid=7ffff2c7d700;cpuset=[5])<o:p></o:p> EAL: lcore 13 is ready (tid=7fffeec75700;cpuset=[13])<o:p></o:p> EAL: lcore 6 is ready (tid=7ffff247c700;cpuset=[6])<o:p></o:p> EAL: Trying to obtain current memory policy.<o:p></o:p> EAL: Setting policy MPOL_PREFERRED for socket 0<o:p></o:p> EAL: Restoring previous memory policy: 0<o:p></o:p> EAL: request: mp_malloc_sync<o:p></o:p> EAL: Heap on socket 0 was expanded by 1024MB<o:p></o:p> EAL: PCI device 0000:0b:00.0 on NUMA socket -1<o:p></o:p> EAL: probe driver: 15b3:1018 net_mlx5<o:p></o:p> EAL: Mem event callback 'MLX5_MEM_EVENT_CB:(nil)' registered<o:p></o:p> net_mlx5: checking device "mlx5_0"<o:p></o:p> net_mlx5: PCI information matches for device "mlx5_0"<o:p></o:p> net_mlx5: no E-Switch support detected<o:p></o:p> net_mlx5: naming Ethernet device "0b:00.0"<o:p></o:p> net_mlx5: DevX is NOT supported<o:p></o:p> net_mlx5: enhanced MPW is supported<o:p></o:p> net_mlx5: SWP support: 0<o:p></o:p> net_mlx5: min_single_stride_log_num_of_bytes: 6<o:p></o:p> net_mlx5: max_single_stride_log_num_of_bytes: 13<o:p></o:p> net_mlx5: min_single_wqe_log_num_of_strides: 9<o:p></o:p> net_mlx5: max_single_wqe_log_num_of_strides: 16<o:p></o:p> net_mlx5: supported_qpts: 256<o:p></o:p> net_mlx5: device supports Multi-Packet RQ<o:p></o:p> net_mlx5: tunnel offloading is supported<o:p></o:p> net_mlx5: MPLS over GRE/UDP tunnel offloading is not supported<o:p></o:p> net_mlx5: checksum offloading is supported<o:p></o:p> net_mlx5: maximum Rx indirection table size is 512<o:p></o:p> net_mlx5: VLAN stripping is supported<o:p></o:p> net_mlx5: FCS stripping configuration is supported<o:p></o:p> net_mlx5: enhanced MPS is enabled<o:p></o:p> net_mlx5: port 0 MAC address is 00:50:56:a9:a9:3e<o:p></o:p> net_mlx5: port 0 MTU is 2040<o:p></o:p> net_mlx5: port 0 forcing Ethernet interface up<o:p></o:p> net_mlx5: min tx inline configured: 0<o:p></o:p> net_mlx5: Hash list with mlx5_0_flow_table size 0x1000 is created.<o:p></o:p> net_mlx5: Hash list with mlx5_0_tags size 0x2000 is created.<o:p></o:p> net_mlx5: port 0 flow maximum priority: 3<o:p></o:p> net_mlx5: metadata mode 0<o:p></o:p> net_mlx5: metadata MARK mask 00FFFFFF<o:p></o:p> net_mlx5: metadata META mask FFFFFFFF<o:p></o:p> net_mlx5: metadata reg_c0 mask FFFFFFFF<o:p></o:p> net_mlx5: port 0 extensive metadata register is not supported<o:p></o:p> net_mlx5: Unexpected error in DR drop action support detection<o:p></o:p> EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)<o:p></o:p> Succeeded to init eal<o:p></o:p> net_mlx5: port 0 Tx queues number update: 0 -> 1<o:p></o:p> net_mlx5: port 0 Rx queues number update: 0 -> 1<o:p></o:p> Succeeded to configure port<o:p></o:p> net_mlx5: port 0 adapter MTU set to 2040<o:p></o:p> Succeeded to set mtu port<o:p></o:p> net_mlx5: port 0 configuring Rx queue 0 for 512 descriptors<o:p></o:p> net_mlx5: port 0 maximum number of segments per packet: 1<o:p></o:p> net_mlx5: port 0 CRC stripping is disabled, 4 bytes will be subtracted from incoming frames to hide it<o:p></o:p> net_mlx5: port 0 adding Rx queue 0 to list<o:p></o:p> Succeeded to rx queue setup queue 0<o:p></o:p> testReceiver.cpp 103<o:p></o:p> net_mlx5: port 0 configuring queue 0 for 512 descriptors<o:p></o:p> net_mlx5: port 0 adding Tx queue 0 to list<o:p></o:p> Succeeded to tx queue setup queue 0<o:p></o:p> net_mlx5: port 0 starting device<o:p></o:p> net_mlx5: port 0 Rx queues number update: 1 -> 1<o:p></o:p> net_mlx5: port 0 Tx queue 0 allocated and configured 512 WRs<o:p></o:p> net_mlx5: port 0: uar_mmap_offset 0x306000<o:p></o:p> net_mlx5: port 0 Rx queue 0 registering mp pool having 1 chunks<o:p></o:p> net_mlx5: port 0 creating a MR using address (0x15a9fd6c0)<o:p></o:p> net_mlx5: device mlx5_0 inserting MR(0x15a9f4ec0) to global cache<o:p></o:p> net_mlx5: inserted B-tree(0x17ffdcb60)[1], [0x140000000, 0x180000000) lkey=0x7a0d0100<o:p></o:p> net_mlx5: inserted B-tree(0x15a9fbfec)[1], [0x140000000, 0x180000000) lkey=0x7a0d0100<o:p></o:p> net_mlx5: port 0 Rx queue 0 allocated and configured 512 segments (max 512 packets)<o:p></o:p> net_mlx5: port 0 device_attr.max_qp_wr is 32768<o:p></o:p> net_mlx5: port 0 device_attr.max_sge is 30<o:p></o:p> net_mlx5: port 0 rxq 0 updated with 0x7fffffffd688<o:p></o:p> net_mlx5: port 0 failed to set defaults flows<o:p></o:p> net_mlx5: port 0 Rx queue 0 freeing WRs<o:p></o:p> DPDK: Error starting or stopping port -12<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Asaf Penso <asafp@nvidia.com> Sent: Friday, April 1, 2022 9:39 AM To: Yaron Illouz <yaroni@radcom.com>; users@dpdk.org; dev@dpdk.org Subject: Re: dpdk rte rss flow with sriov<o:p></o:p> </div> </div> <o:p> </o:p> <div> I understand your scenario. <o:p></o:p> </div> <div> I want referring to action VF.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> Can you share the testpmd commands line you use in the VM?<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div id="ms-outlook-mobile-signature"> <div> Regards,<o:p></o:p> </div> <div> Asaf Penso<o:p></o:p> </div> </div> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="2" width="98%" align="center"> </div> <div id="divRplyFwdMsg"> From: Yaron Illouz <<a href="mailto:yaroni@radcom.com">yaroni@radcom.com</a>> Sent: Tuesday, March 29, 2022 5:33:39 PM To: Asaf Penso <<a href="mailto:asafp@nvidia.com">asafp@nvidia.com</a>>; <a href="mailto:users@dpdk.org"> users@dpdk.org</a> <<a href="mailto:users@dpdk.org">users@dpdk.org</a>>; <a href="mailto:dev@dpdk.org"> dev@dpdk.org</a> <<a href="mailto:dev@dpdk.org">dev@dpdk.org</a>> Subject: RE: dpdk rte rss flow with sriov <o:p></o:p> <div> <o:p></o:p> </div> </div> <div> <div> Perhaps I didn’t ask correctly<o:p></o:p> My vm have a vf sriov, <o:p></o:p> The dpdk application will create x rx queues on the vf as it should have done for a pf<o:p></o:p> I expect traffic to be spreaded between the rx queues according to rte_flow rss defined (RTE_FLOW_ACTION_TYPE_QUEUE or RTE_FLOW_ACTION_TYPE_RSS)<o:p></o:p> <p class="xmsonormal" style="font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:inherit;box-sizing:inherit;orphans:2;widows:2;text-decoration-style:initial;text-decoration-color:initial;word-spacing:0px"> Asaf: I am not asking about the RTE_FLOW_ACTION_TYPE_VF – When you said Mellanox support where you referring to RTE_FLOW_ACTION_TYPE_VF?<o:p></o:p> <o:p></o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Asaf Penso <<a href="mailto:asafp@nvidia.com">asafp@nvidia.com</a>> Sent: Tuesday, March 29, 2022 4:40 PM To: Yaron Illouz <<a href="mailto:yaroni@radcom.com">yaroni@radcom.com</a>>; <a href="mailto:users@dpdk.org">users@dpdk.org</a>; <a href="mailto:dev@dpdk.org"> dev@dpdk.org</a> Subject: RE: dpdk rte rss flow with sriov<o:p></o:p> </div> </div> <o:p></o:p> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="background:#FFF2CC;border-collapse:collapse"> <tbody> <tr> <td width="576" valign="top" style="width:432.35pt;background:#FBE4D5;padding:0cm 5.4pt 0cm 5.4pt"> <a name="x__Hlk88152779">EXTERNAL EMAIL:</a> Do not click links or attachments unless you recognize the sender and know the content is safe<o:p></o:p> </td> </tr> </tbody> </table> <o:p></o:p> <div> Mellanox NICs support RSS for PF/VF/SF.<o:p></o:p> <o:p></o:p> <div> Regards,<o:p></o:p> Asaf Penso<o:p></o:p> </div> <o:p></o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Yaron Illouz <<a href="mailto:yaroni@radcom.com">yaroni@radcom.com</a>> Sent: Tuesday, March 29, 2022 4:35 PM To: <a href="mailto:users@dpdk.org">users@dpdk.org</a>; <a href="mailto:dev@dpdk.org"> dev@dpdk.org</a> Subject: dpdk rte rss flow with sriov<o:p></o:p> </div> </div> <o:p></o:p> Hi <o:p></o:p> <o:p></o:p> Is this possible to use dpdk rte rss flow with a vf sriov, or is it an option available only for real interface?<o:p></o:p> I use dpdk 19.11 and a Mellanox card 100G with mlx5<o:p></o:p> </div> </div> </div> </div> </div> </body> </html>