<div dir="ltr"> I hit the same issue, and it was brute force. The first version of VMXNET3 is something that has never been exposed directly. (It is implicit and based on the VM Hardware version.A) You can adjust it in the VM advanced settings. It makes sense to indicate exactly which flag must be in the VM. (Why? Sometimes, downgrading the VM Hardware version is not an option. ) While downgrading, the VMXNET3 version will run in compatibility mode. (I.e., DPDK version old, nic new, RSS must X, etc, etc) There are a few combinations where You can not assume a single queue, or take a four-queue, or assume a 32-queue. (This combination depends on Combination NIC - VM - ESXi - DPDK version So, asking a client to do four variations and find a combination is not a good option. (Note in kernel driver it's the same, you can use 32, but you have no idea how to enable the VMXNET3 version that will allow you to use 32 queues) Example. You have 810 - VMXNET 3 poll mode and enable 16 queues. What do you do? You have 810 - VMXNET 3 (the default queue in RSS 4). What do you do? --- This will fail on 23/24/25. You have 801 - VMXNET 3 - ESXi 8.0.2 and DPDK 22.xx app ( example people use trex it build 22 or 23)<div> I just gave a few examples. Kind Regards, Mus <div> </div></div></div> <div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Sat, Apr 19, 2025 at 11:15 AM Fangyin Hu <<a href="mailto:FHu@sonicwall.com">FHu@sonicwall.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-7072844246104021192"> <div lang="EN-US" style="overflow-wrap: break-word;"> <div class="m_-7072844246104021192WordSection1"> Hi, Knight Thank you very much for your reply! I check our codes, my application config the 1 RX queue for vmxnet3 device, so the call vmxnet3_v4_rss_configure() will failed due to the single RX queue. I adjusted our codes and use 2 RX queue for vmxnet3 device, then retest my application aga in. Now my application works fine and here are some debug logs. [2025-04-18 07:37:02]:: Try to configure port 0, RX Queue Number: 2, TX Queue Number: 1 Dev 0^M^M [2025-04-18 07:37:02]:: pktio/dpdk.c:310:init_options():DPDK interface (net_vmxnet3): 0^M^M [2025-04-18 07:37:02]:: pktio/dpdk.c:312:init_options(): multicast_en: 1^M^M [2025-04-18 07:37:02]:: pktio/dpdk.c:313:init_options(): num_rx_desc: 4096^M^M [2025-04-18 07:37:02]:: pktio/dpdk.c:314:init_options(): num_tx_desc: 512^M^M [2025-04-18 07:37:02]:: pktio/dpdk.c:315:init_options(): rx_drop_en: 0^M^M [2025-04-18 07:37:02]:: odp_packet_io.c:439:odp_pktio_open():interface: 0, driver: dpdk^M^M [2025-04-18 07:37:02]:: created pktio 1 idx( 0) (0)^M^M [2025-04-18 07:37:02]:: ### starting data port pktio's...^M^M [2025-04-18 07:37:02]:: pktio/dpdk.c:2277:dpdk_setup_eth_tx():Mapping per TX queue statistics not supported: -95^M^M [2025-04-18 07:37:02]:: pktio/dpdk.c:2281:dpdk_setup_eth_tx():Mapped 0/16 TX counters^M^M [2025-04-18 07:37:02]:: pktio/dpdk.c:2315:dpdk_setup_eth_rx():Mapping per RX queue statistics not supported: -95^M^M [2025-04-18 07:37:02]:: pktio/dpdk.c:2319:dpdk_setup_eth_rx():Mapped 0/16 RX counters^M^M [2025-04-18 07:37:02]:: --[vmxnet3_dev_start:963] hw->version: 6, rss_hf: 3380 --^M^M [2025-04-18 07:37:02]:: --[vmxnet3_rss_configure:1430] rss_hf: 3380 --^M^M [2025-04-18 07:37:02]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M [2025-04-18 07:37:02]:: odp_packet_io.c:723:odp_pktio_start():interface: 0, input queues: 2, output queues: 1^M^M [2025-04-18 07:37:03]:: pktio port 0 dpdk port 0 started!^M^M So the next step we need to modify our codes, it will not config the RSS when the device RX queue number is 1 Thank you for your advise again and best wish to you! <div> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> From: Knight, Joshua <<a href="mailto:Joshua.Knight@netscout.com" target="_blank">Joshua.Knight@netscout.com</a>> Sent: Wednesday, April 16, 2025 9:59 PM To: Fangyin Hu <FHu@SonicWall.com>; <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> Subject: Re: VMware run the DPDK with vmxnet3 NIC failed with the RSS config </div> </div> <div> <div style="border:1pt solid rgb(0,51,51);padding:2pt"> <div> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. </div> </div> </div> <div> We’ve hit the same problem setting up RSS on vmxnet3 on newer esxi8 releases. In our case we would see the failure if we only had a single rx queue. Snippet from my discussion with their support team: > So to summarize, with VM Hardware version 14 in esxi 7 and 6.7, setting the RSS hash function did not generate an error in the driver. With VM Hardware version 14 in esxi 8U2, it generates an error in the driver. I had a ticket opened with Broadcom and their response was they made a change in behavior where instead of silently rejecting the RSS configuration (since, really you don’t need RSS with a single queue although we still wanted it enabled) the nic now returns an error. The biggest problem I had with the change is they applied it to *earlier* vm hw versions, not just the latest/newest version. Broadcom’s response to my ticket was: >Regarding the difference in behavior you have observed with ESXi versions, there were some enhancements introduced in vSphere version 7.0U1 used along with VM hardware version >=14, it identifies the RSS misconfiguration with Rx Queue and throws an error message similar to what has been observed. In older versions, it was not erroring out so you will not observe any failures in the applications. <div id="m_-7072844246104021192mail-editor-reference-message-container"> <div> <div> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in"> From: Fangyin Hu <<a href="mailto:FHu@SonicWall.com" target="_blank">FHu@SonicWall.com</a>> Date: Wednesday, April 16, 2025 at 4:23 AM To: <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> <<a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a>> Subject: VMware run the DPDK with vmxnet3 NIC failed with the RSS config </div> <div> Hi My application developed with the DPDK, we found out one error message when my application run in the VMware platform. [2025-04-15 03:01:09]:: --[vmxnet3_dev_start:961] hw->version: 6, rss_hf: 3380 --^M^M [2025-04-15 03:01:09]:: --[vmxnet3_rss_configure:1434] rss_hf: 3380 --^M^M [2025-04-15 03:01:09]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M [2025-04-15 03:01:09]:: vmxnet3_v4_rss_configure(): Set RSS fields (v4) failed: 1^M^M [2025-04-15 03:01:09]:: vmxnet3_dev_start(): Failed to configure v4 RSS^M^M From the debug logs, it seems that the vmxnet3 device failed to configure the v4 RSS. We can found these informations from the DPDK’s code snippet: rss_hf = 3380 = 0xd34 So the RSS configure is this: #define VMXNET3_RSS_OFFLOAD_ALL ( \ RTE_ETH_RSS_IPV4 | \ RTE_ETH_RSS_NONFRAG_IPV4_TCP | \ RTE_ETH_RSS_IPV6 | \ RTE_ETH_RSS_NONFRAG_IPV6_TCP) #define VMXNET3_V4_RSS_MASK ( \ RTE_ETH_RSS_NONFRAG_IPV4_UDP | \ RTE_ETH_RSS_NONFRAG_IPV6_UDP) The function call vmxnet3_rss_configure() is works fine but the function call vmxnet3_v4_rss_configure() failed: if (VMXNET3_VERSION_GE_4(hw) && dev->data->dev_conf.rxmode.mq_mode == RTE_ETH_MQ_RX_RSS) { /* Check for additional RSS */ ret = vmxnet3_v4_rss_configure(dev); if (ret != VMXNET3_SUCCESS) { PMD_INIT_LOG(ERR, "Failed to configure v4 RSS"); return ret; } } /* * Additional RSS configurations based on vmxnet v4+ APIs */ int vmxnet3_v4_rss_configure(struct rte_eth_dev *dev) { struct vmxnet3_hw *hw = dev->data->dev_private; Vmxnet3_DriverShared *shared = hw->shared; Vmxnet3_CmdInfo *cmdInfo = &shared->cu.cmdInfo; struct rte_eth_rss_conf *port_rss_conf; uint64_t rss_hf; uint32_t ret; PMD_INIT_FUNC_TRACE(); cmdInfo->setRSSFields = 0; port_rss_conf = &dev->data->dev_conf.rx_adv_conf.rss_conf; printf("--[%s:%d] rss_hf: %llu --\n", __FUNCTION__, __LINE__, port_rss_conf->rss_hf); if ((port_rss_conf->rss_hf & VMXNET3_MANDATORY_V4_RSS) != VMXNET3_MANDATORY_V4_RSS) { PMD_INIT_LOG(WARNING, "RSS: IPv4/6 TCP is required for vmxnet3 v4 RSS," "automatically setting it"); port_rss_conf->rss_hf |= VMXNET3_MANDATORY_V4_RSS; } rss_hf = port_rss_conf->rss_hf & (VMXNET3_V4_RSS_MASK | VMXNET3_RSS_OFFLOAD_ALL); if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_TCP) cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP4; if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_TCP) cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP6; if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_UDP) cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP4; if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_UDP) cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP6; VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD, VMXNET3_CMD_SET_RSS_FIELDS); ret = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD); if (ret != VMXNET3_SUCCESS) { PMD_DRV_LOG(ERR, "Set RSS fields (v4) failed: %d", ret); } return ret; } It seems like the VMXNET3_WRITE_BAR1_REG failed, but don’t know why this happened. My DPDK version is 22.11.1 and the vmxnet3 driver version is 1.6.0.0-k-NAPI BTW we just upgraded VM compatibility to ESXi 8.0 U2 and later (VM version 21), then our application run with this error logs. The previous VM compatibility is ESXi 6.5 and later (VM version 13), and our application works fine. It is because the vmxnet3 hw version is 3, so it will not trigger to call the V4 RSS configure. Could someone help to check this issue? Or maybe I missed something? </div> </div> </div> </div> </div> </div> </div> </div></blockquote></div>