<div dir="ltr">Hi Peter<div>That's great!<br><div>The queue-format is determined internally based on the VM instance. All gen2 (like n2, e2, g2 etc.) use GQI_QPL format. There is no way for the customer to configure it.</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 30, 2024 at 6:50 AM Peter Morrow <<a href="mailto:peter@graphiant.com" target="_blank">peter@graphiant.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div> <div dir="ltr"> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Thanks Rushil,</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> I tested with 23.11 and everything is good now. Out of interest, what determines the queue format on the device? Is there a way to change the queue format such that 22.11 where gve support first appear could be used?</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Thanks,</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Peter.</div> <div id="m_-3835003080218866411m_-3159456119156087190appendonsend"></div> <hr style="display:inline-block;width:98%"> <div id="m_-3835003080218866411m_-3159456119156087190divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Rushil Gupta <<a href="mailto:rushilg@google.com" target="_blank">rushilg@google.com</a>><br> <b>Sent:</b> 29 October 2024 21:14<br> <b>To:</b> Peter Morrow <<a href="mailto:peter@graphiant.com" target="_blank">peter@graphiant.com</a>><br> <b>Cc:</b> <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> <<a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a>>; <a href="mailto:joshwash@google.com" target="_blank">joshwash@google.com</a> <<a href="mailto:joshwash@google.com" target="_blank">joshwash@google.com</a>>; <a href="mailto:jeroendb@google.com" target="_blank">jeroendb@google.com</a> <<a href="mailto:jeroendb@google.com" target="_blank">jeroendb@google.com</a>>; <a href="mailto:jungeng.guo@intel.com" target="_blank">jungeng.guo@intel.com</a> <<a href="mailto:jungeng.guo@intel.com" target="_blank">jungeng.guo@intel.com</a>><br> <b>Subject:</b> Re: gve queue format on dpdk 22.11</font> <div> </div> </div> <div> <p></p> <div style="border:1pt solid rgb(156,101,0);padding:1pt;background-color:rgb(255,249,102)"> <p style="color:red;text-align:center"><span>This email originated outside of your organization, do not visit any link, open any attachments or provide any sensitive information unless you recognize the sender and are certain the content can be trusted</span> </p> </div> <p></p> <div> <div dir="ltr">Hi Peter <div>DQO_RDA is supported 23.07 onwards.</div> </div> <br> <div> <div dir="ltr">On Tue, Oct 29, 2024 at 10:09 AM Peter Morrow <<a href="mailto:peter@graphiant.com" target="_blank">peter@graphiant.com</a>> wrote:<br> </div> <blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div> <div dir="ltr"> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Hi Folks,</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Reading the docs for the the 22.11 release it clearly states that the DQO_RDA queue format is not yet supported:<br> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <a href="https://doc.dpdk.org/guides-22.11/nics/gve.html" id="m_-3835003080218866411m_-3159456119156087190x_m_-5908874388268627887m_8575338673047331922LPlnk936366" target="_blank">https://doc.dpdk.org/guides-22.11/nics/gve.html</a></div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> I'm attempting to bring up our software router on GCP (VM instance type c4-standard-8) where we are currently using dpdk 22.11 (via debian, with a 6.1.0-26-amd64 kernel), given the lack of support for DQO_RDA I see the following expected messages when I start vpp (our dpdk application):</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Oct 29 11:11:59 gcpt2 vnet[1919]: dpdk: gve_adminq_describe_device(): Driver is running with DQO RDA queue format.</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Oct 29 11:11:59 gcpt2 vnet[1919]: dpdk: gve_adminq_describe_device(): MAC addr: 42:01:0A:07:00:0F</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Oct 29 11:11:59 gcpt2 vnet[1919]: dpdk: gve_init_priv(): Max TX queues 2, Max RX queues 2</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Oct 29 11:11:59 gcpt2 vnet[1919]: dpdk: gve_dev_init(): DQO_RDA is not implemented and will be added in the future</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> As a result vpp sefaults shortly after:</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> (gdb) bt</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #0 gve_adminq_create_tx_queue (queue_index=0, priv=0xbc03a59c0) at ../drivers/net/gve/base/gve_adminq.c:502</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #1 gve_adminq_create_tx_queues (priv=0xbc03a59c0, num_queues=2) at ../drivers/net/gve/base/gve_adminq.c:516</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #2 0x00007ffda6c9c5d6 in gve_dev_start (dev=0x7ffe1f4f7240 <rte_eth_devices>) at ../drivers/net/gve/gve_ethdev.c:181</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #3 0x00007ffe1f489b6a in rte_eth_dev_start () from /lib/x86_64-linux-gnu/librte_ethdev.so.23</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #4 0x00007ffff584aa14 in dpdk_device_start (xd=xd@entry=0x7ffe204fbac0) at ./src/plugins/dpdk/device/common.c:393</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #5 0x00007ffff584c598 in dpdk_interface_admin_up_down (vnm=<optimized out>, hw_if_index=<optimized out>, flags=<optimized out>)</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> at ./src/plugins/dpdk/device/device.c:485</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #6 0x00007ffff6ab3ccd in vnet_sw_interface_set_flags_helper (vnm=vnm@entry=0x7ffff7f6b660 <vnet_main>, sw_if_index=<optimized out>, </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP, helper_flags=0, helper_flags@entry=VNET_INTERFACE_SET_FLAGS_HELPER_WANT_REDISTRIBUTE)</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> at ./src/vnet/interface.c:545</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #7 0x00007ffff6ab489f in vnet_sw_interface_set_flags (vnm=vnm@entry=0x7ffff7f6b660 <vnet_main>, sw_if_index=<optimized out>, </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> flags=<optimized out>) at ./src/vnet/interface.c:601</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #8 0x00007ffff6accee9 in vl_api_sw_interface_set_flags_t_handler (mp=0x7ffe3f241ff8) at ./src/vnet/interface_api.c:100</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #9 0x00007ffff7fa6e0d in msg_handler_internal (free_it=0, do_it=1, trace_it=<optimized out>, msg_len=<optimized out>, </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> the_msg=0x7ffe3f241ff8, am=0x7ffff7fb8f40 <api_global_main>) at ./src/vlibapi/api_shared.c:580</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #10 vl_msg_api_handler_no_free (the_msg=0x7ffe3f241ff8, msg_len=<optimized out>) at ./src/vlibapi/api_shared.c:652</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #11 0x00007ffff7f8ed7f in vl_socket_process_api_msg (rp=<optimized out>, input_v=<optimized out>) at ./src/vlibmemory/socket_api.c:208</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #12 0x00007ffff7f978d3 in vl_api_clnt_process (vm=<optimized out>, node=<optimized out>, f=<optimized out>)</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> at ./src/vlibmemory/memclnt_api.c:429</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #13 0x00007ffff6853966 in vlib_process_bootstrap (_a=<optimized out>) at ./src/vlib/main.c:1223</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #14 0x00007ffff67ba03c in clib_calljmp () at /usr/src/packages/BUILD/src/vppinfra/longjmp.S:123</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #15 0x00007ffe1edfcd90 in ?? ()</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #16 0x00007ffff6855054 in vlib_process_startup (f=0x0, p=0x7ffe2078f780, vm=0x7ffe1fa007c0) at ./src/vlib/main.c:1248</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #17 dispatch_process (vm=0x7ffe1fa007c0, p=<optimized out>, last_time_stamp=<optimized out>, f=0x0) at ./src/vlib/main.c:1304</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> #18 0x0000000000000000 in ?? ()</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> (gdb)</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> The segfault occurs due to the complq pointer here being NULL:</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> cmd.create_tx_queue.tx_comp_ring_addr =</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> cpu_to_be64(txq->complq->tx_ring_phys_addr);</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> It may be possible to upgrade to a more recent version of dpdk which should help me progress, though before I do that I wondered if there is any other way to make progress here? Specifically, is there a way to force a different queue format in the device? This is a c4-standard-8 VM running in GCE.<br> <br> Thanks!<br> Peter.</div> <div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> </div> </div> </div> </blockquote> </div> </div> </div> </div> </div></blockquote></div>