<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size:11.0pt">Which FW and OFED version are you using?<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt">BTW, have you tried on 6-Dx and can you try on the non-root (>0) group? I remember of the send_to_kernel is only supported on non-root tables.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Gopinath K <gopinath.gk@zohomail.com> <br> <b>Sent:</b> Tuesday, November 5, 2024 4:23 PM<br> <b>To:</b> users <users@dpdk.org><br> <b>Subject:</b> Error with SEND_TO_KERNEL Action during Flow creation on Mellanox CX-5<o:p></o:p></span></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <table class="MsoNormalTable" border="1" cellpadding="0" style="background:#FFEB9C"> <tbody> <tr> <td style="padding:.75pt .75pt .75pt .75pt"> <p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black"> </span><o:p></o:p></p> </td> </tr> </tbody> </table> <p class="MsoNormal"><o:p> </o:p></p> <div> <div> <div> <div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Hi,<br> <br> I'm new to DPDK, and have been working on creating flow rules by using DPDK, to steer traffic to the kernel.<br> <br> To understand the functionality of the Mellanox Bifuracted drive, i attempted to steer ICMP packets to Linux Kernel and other traffics to DPDK application. first i tried to create simple flow rule which drops the packet whose destination equals 192.168.3.2 <a href="https://doc.dpdk.org/guides/howto/rte_flow.html" target="_blank">(refrenced from from API examples)</a> the flow created successfully without an error.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> However the issue arises when I attempt to use the RTE_FLOW_ACTION_TYPE_SEND_TO_KERNEL, which results in an error.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Below the sample code.<br> <br> 1. functions to validate and create flow rule.<o:p></o:p></span></p> </div> <div> <blockquote style="border:solid #CCCCCC 1.0pt;padding:5.0pt 5.0pt 5.0pt 5.0pt;margin-top:5.0pt;margin-bottom:5.0pt"> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">static ::rte_flow* create_flow(uint16_t port_id, rte_flow_attr& attr, rte_flow_item& pattern, rte_flow_action& actions) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_error error;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> auto flow = rte_flow_create(port_id, &attr, &pattern, &actions, &error);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (!flow) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[create flow rule] failed to create flow rule, error: %s.\n", error.message);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return flow;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">}</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">bool validate_flow(uint16_t port_id, rte_flow_attr& attr, rte_flow_item& pattern, rte_flow_action& action) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_error error;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> auto res = rte_flow_validate(port_id, &attr, &pattern, &action, &error);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (res < 0) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[validate flow rule] validate flow failed, error: %s\n", error.message);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return false;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return true;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">}</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> </blockquote> </div> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> 2. function to enable isolated mode on network port.<o:p></o:p></span></p> </div> <div> <blockquote style="border:solid #CCCCCC 1.0pt;padding:5.0pt 5.0pt 5.0pt 5.0pt;margin-top:5.0pt;margin-bottom:5.0pt"> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">bool enable_isolated_mode(uint16_t port_id) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_error error;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> // enable isolated mode.</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (rte_flow_isolate(port_id, 1, &error) < 0) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[isolated mode] failed to enable isolated mode on port %u, error: %s\n", port_id, error.message);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return false;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[isolated mode] isolated mode enabled on port %u\n", port_id);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return true;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">}</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> </blockquote> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> 3. port initialization function.<o:p></o:p></span></p> </div> <blockquote style="border:solid #CCCCCC 1.0pt;padding:5.0pt 5.0pt 5.0pt 5.0pt;margin-top:5.0pt;margin-bottom:5.0pt"> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">static inline int</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">port_initilization(uint16_t port, rte_mempool *mbuf_pool) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_eth_conf port_conf;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (!rte_eth_dev_is_valid_port(port)) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[port init] port %u is not a valid port\n", port);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return -1;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (!enable_isolated_mode(port)) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return -1;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> memset(&port_conf, 0, sizeof(rte_eth_conf));</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_eth_dev_info device_info;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> auto retval = rte_eth_dev_info_get(port, &device_info);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (retval != 0) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[pirt init] getting device(port %u) info, error: %d [%s]\n", port, rte_errno, ::rte_strerror(rte_errno));</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return retval;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (device_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE)</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> port_conf.txmode.offloads |= RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> uint16_t rx_rings = 1;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> uint16_t tx_rings = 1;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (retval != 0) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[port init] failed to configure port(%u), error: %d [%s]\n", port, rte_errno, ::rte_strerror(rte_errno));</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return retval;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> uint16_t nb_rxd = RX_RING_SIZE;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> uint16_t nb_txd = TX_RING_SIZE;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> retval = rte_eth_dev_adjust_nb_rx_tx_desc(port, &nb_rxd, &nb_txd);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (retval != 0) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[port init] rte_eth_dev_adjust failed, error: %d [%s]\n", rte_errno, ::rte_strerror(rte_errno));</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return retval;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> for (auto q = 0; q < rx_rings; q++) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> retval = rte_eth_rx_queue_setup(port, q, nb_rxd, rte_eth_dev_socket_id(port), NULL, mbuf_pool);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (retval < 0)</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return retval;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> auto tx_conf = device_info.default_txconf;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> tx_conf.offloads = port_conf.txmode.offloads;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> for (auto q = 0; q < tx_rings; q++) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> retval = rte_eth_tx_queue_setup(port, q, nb_txd, rte_eth_dev_socket_id(port), &tx_conf);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (retval < 0)</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return retval;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> ::rte_ether_addr addr;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> retval = rte_eth_macaddr_get(port, &addr);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (retval != 0)</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return retval;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> retval = rte_eth_dev_start(port);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (retval != 0) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[port init] starting port %u failed, error: %d [%s]\n", port, rte_errno, rte_strerror(rte_errno));</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return retval;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[port init] successfully started port: %u\n", port);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("port %u MAC: %02" PRIx8 " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 "\n", port, RTE_ETHER_ADDR_BYTES(&addr));</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> init_ingress_flow_rules(port);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> init_egress_flow_rule(port);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> return 0;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">}</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> </blockquote> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> 4. Flow rule<o:p></o:p></span></p> </div> </div> <div> <blockquote style="border:solid #CCCCCC 1.0pt;padding:5.0pt 5.0pt 5.0pt 5.0pt;margin-top:5.0pt;margin-bottom:5.0pt"> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">static void init_ingress_flow_rule(uint32_t port_id) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_attr attr = {0};</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_item pattern[4];</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_action action[2];</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> attr.ingress = 1;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> memset(pattern, 0, sizeof pattern);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_item_eth eth_spec;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_item_eth eth_mask;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> memset(&eth_spec, 0, sizeof eth_spec);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> memset(&eth_mask, 0, sizeof eth_mask);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> pattern[0].type = RTE_FLOW_ITEM_TYPE_ETH;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> pattern[0].spec = &eth_spec;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_item_ipv4 ipv4_spec;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_item_ipv4 ipv4_mask;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> memset(&ipv4_spec, 0, sizeof ipv4_spec);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> memset(&ipv4_mask, 0, sizeof ipv4_mask);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> pattern[1].type = RTE_FLOW_ITEM_TYPE_IPV4;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> pattern[1].spec = &ipv4_spec;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> pattern[1].mask = &ipv4_mask;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_item_icmp icmp_spec;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> rte_flow_item_icmp icmp_mask;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> memset(&icmp_spec, 0, sizeof icmp_spec);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> memset(&icmp_mask, 0, sizeof icmp_mask);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> icmp_spec.hdr.icmp_type = 0;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> pattern[2].type = RTE_FLOW_ITEM_TYPE_ICMP;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> pattern[2].spec = &icmp_spec;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> pattern[2].mask = &icmp_mask;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> pattern[3].type = RTE_FLOW_ITEM_TYPE_END;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> action[0].type = RTE_FLOW_ACTION_TYPE_SEND_TO_KERNEL;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> action[1].type = RTE_FLOW_ACTION_TYPE_END;</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (validate_flow(port_id, attr, pattern[0], action[0])) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[init flow rule] validated successfully.\n");</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> auto flow = create_flow(port_id, attr, pattern[0], action[0]);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> if (flow != NULL) {</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> printf("[rte flow rule] successfully created a egress flow rule in port %u\n", port_id);</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"> }</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">}</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> </blockquote> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">When I try to run the above program, the flow rule validated successfully with `rte_flow_validate()`, but failed on creating the flow with an error of `hardware refuses to create flow`.<o:p></o:p></span></p> </div> <blockquote style="border:solid #CCCCCC 1.0pt;padding:5.0pt 5.0pt 5.0pt 5.0pt;margin-top:5.0pt;margin-bottom:5.0pt"> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">[init flow rule] validated successfully.</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">[create flow rule] failed to create flow rule, error: hardware refuses to create flow.</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> </blockquote> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">The same error arise when try to create the flow in `dpdk-testpmd` program.<o:p></o:p></span></p> </div> <blockquote style="border:solid #CCCCCC 1.0pt;padding:5.0pt 5.0pt 5.0pt 5.0pt;margin-top:5.0pt;margin-bottom:5.0pt"> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">testpmd> flow isolate 0 true</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">testpmd> port start 0</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span class="size"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black;background:white">testpmd> flow create 0 ingress pattern eth / ipv4 / icmp / end actions send_to_kernel / end</span></span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="background:white"><span class="size"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black;background:white">port_flow_complain(): Caught PMD error type 1 (cause unspecified): hardware refuses to create flow: Operation not supported</span></span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black"><br> Can anyone tell me whether my code is correct or not?</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> </blockquote> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Can anyone tell me whether my code is correct or not? and any idea how to use SEND_TO_KERNEL Action on Mellanox CX-5 NIC's.<br> <br> System specification.<br> <br> Operating System: Debian<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> OS Version: 12.7<br> DPDK version: 24.07<br> <br> <br> Thanks and Regards,<br> Gopinath.<o:p></o:p></span></p> </div> </div> <div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> </div> <div> <div> <div> <div id=""> <div> <p><span class="colour"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#2A2A2A">Sent using Zoho Mail</span></span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> </div> </div> </div> </div> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> </div> </div> </div> </body> </html>