<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="en-DE" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hello Gregory,<o:p></o:p> <o:p> </o:p> Thanks for suggesting the workaround. Looking forward to the investigation result and the fix.<o:p></o:p> <o:p> </o:p> Regards,<o:p></o:p> Tao<o:p></o:p> <o:p> </o:p> <div id="mail-editor-reference-message-container"> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Gregory Etelson <getelson@nvidia.com> Date: Wednesday, 20. March 2024 at 17:36 To: Tao Li <byteocean@hotmail.com>, Suanming Mou <suanmingm@nvidia.com>, guvenc.gulce@gmail.com <guvenc.gulce@gmail.com>, users@dpdk.org <users@dpdk.org> Subject: Re: mlx5: rte_flow template/async API raw_encap validation bug ?<o:p></o:p> </div> <div> Hello Tao,<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> I reproduced the PMD crash you've described.<o:p></o:p> </div> <div> We'll investigate it and will issue a fix shortly.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> In the meanwhile, I can suggest a workaround.<o:p></o:p> </div> <div> Please consider creating actions template with the fully masked RAW_ENCAP action. The fully masked RAW_ENCAP provides data and size parameters in the action description and sets non-zero values in the action mask configuration. Testpmd commands are:<o:p></o:p> </div> <div> dpdk-testpmd -a $PCI,dv_flow_en=2,representor=vf\[0-1\] -- -i<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> port stop all<o:p></o:p> </div> flow configure 0 queues_number 4 queues_size 64<o:p></o:p> flow configure 1 queues_number 4 queues_size 64<o:p></o:p> flow configure 2 queues_number 4 queues_size 64<o:p></o:p> port start all<o:p></o:p> set verbose 1<o:p></o:p> <o:p> </o:p> set raw_decap 0 eth / ipv6 / end_set<o:p></o:p> set raw_encap 0 eth src is 11:22:33:44:55:66 dst is aa:bb:cc:dd:ee:aa type is 0x0800 has_vlan is 0 / end_set<o:p></o:p> <o:p> </o:p> flow actions_template 0 create transfer actions_template_id 1 template raw_decap / raw_encap index 0 / represented_port / end mask raw_decap / raw_encap index 0 / represented_port / end<o:p></o:p> flow pattern_template 0 create transfer pattern_template_id 1 template eth / ipv6 / end<o:p></o:p> flow template_table 0 create transfer table_id 1 group 0 priority 0 rules_number 1 pattern_template 1 actions_template 1<o:p></o:p> <div> <o:p> </o:p> </div> <div> <o:p> </o:p> </div> <div> Regards,<o:p></o:p> </div> <div> Gregory<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="0" width="100%" align="center"> </div> <div> From: Tao Li <byteocean@hotmail.com> Sent: Wednesday, March 20, 2024 17:19 To: Gregory Etelson <getelson@nvidia.com>; Suanming Mou <suanmingm@nvidia.com>; guvenc.gulce@gmail.com <guvenc.gulce@gmail.com>; users@dpdk.org <users@dpdk.org> Subject: Re: mlx5: rte_flow template/async API raw_encap validation bug ?<o:p></o:p> </div> <div> <o:p></o:p> </div> <table class="MsoNormalTable" border="0" cellpadding="0" style="background:#FFEB9C"> <tbody> <tr> <td style="padding:.75pt .75pt .75pt .75pt"> External email: Use caution opening links or attachments<o:p></o:p> </td> </tr> </tbody> </table> <o:p> </o:p> Hello Gregory,<o:p></o:p> <o:p></o:p> I am the colleague from Guevenc. Thanks a lot for providing detailed explaination, and we appreciate your support very much.<o:p></o:p> The guidance on the usage of the RAW_ENCAP mask is adopted and experimented, which currently leads to the segmentation fault in our setup. The example code used to create the action template and table template is as following:<o:p></o:p> <o:p></o:p> <Code Excerpt><o:p></o:p> // first action template<o:p></o:p> struct rte_flow_action_raw_decap decap_action = {.size = (sizeof(struct rte_ether_hdr)+sizeof(struct rte_ipv6_hdr))}; // remove IPinIP packet�s header<o:p></o:p> struct rte_flow_action_raw_encap encap_action = {.data = NULL, .size = sizeof(struct rte_ether_hdr)}; // add ether header for VMs<o:p></o:p> <o:p></o:p> struct rte_flow_action act[] = {<o:p></o:p> [0] = {.type = RTE_FLOW_ACTION_TYPE_RAW_DECAP, .conf = &decap_action}, //?<o:p></o:p> [1] = {.type = RTE_FLOW_ACTION_TYPE_RAW_ENCAP, .conf = &encap_action}, //?<o:p></o:p> [2] = {.type = RTE_FLOW_ACTION_TYPE_REPRESENTED_PORT,},<o:p></o:p> [3] = {.type = RTE_FLOW_ACTION_TYPE_END,},<o:p></o:p> };<o:p></o:p> <o:p></o:p> struct rte_flow_action msk[] = {<o:p></o:p> [0] = {.type = RTE_FLOW_ACTION_TYPE_RAW_DECAP, .conf= &decap_action},<o:p></o:p> [1] = {.type = RTE_FLOW_ACTION_TYPE_RAW_ENCAP, .conf= &encap_action},<o:p></o:p> [2] = {.type = RTE_FLOW_ACTION_TYPE_REPRESENTED_PORT,},<o:p></o:p> [3] = {.type = RTE_FLOW_ACTION_TYPE_END,},<o:p></o:p> };<o:p></o:p> <o:p></o:p> port_template_info_pf.actions_templates[0] = create_actions_template(main_eswitch_port, act, msk);<o:p></o:p> <o:p></o:p> // create template table<o:p></o:p> port_template_info_pf.template_table = create_table_template(main_eswitch_port, &table_attr_pf,<o:p></o:p> (struct rte_flow_pattern_template **)&port_template_info_pf.pattern_templates, MAX_NR_OF_PATTERN_TEMPLATE,<o:p></o:p> (struct rte_flow_actions_template **)&port_template_info_pf.actions_templates, MAX_NR_OF_ACTION_TEMPLATE);<o:p></o:p> <o:p></o:p> </Code Excerpt><o:p></o:p> <o:p></o:p> Using gdb, the following segfault trace is captured:<o:p></o:p> <o:p></o:p> <Trace Excerpt><o:p></o:p> #0 0x00005555579bfb62 in mlx5dr_action_prepare_decap_l3_data (src=0x0, dst=0x7fffffffb3cc "", num_of_actions=6) at ../drivers/net/mlx5/hws/mlx5dr_action.c:2774<o:p></o:p> #1 0x00005555579c2136 in mlx5dr_action_handle_tunnel_l3_to_l2 (action=0x55555f089dc0, num_of_hdrs=1 '\001', hdrs=0x7fffffffb720, log_bulk_sz=1) at ../drivers/net/mlx5/hws/mlx5dr_action.c:1468<o:p></o:p> #2 0x00005555579bd56f in mlx5dr_action_create_reformat_hws (action=0x55555f089dc0, num_of_hdrs=1 '\001', hdrs=0x7fffffffb720, bulk_size=1) at ../drivers/net/mlx5/hws/mlx5dr_action.c:1537<o:p></o:p> #3 0x00005555579bd0eb in mlx5dr_action_create_reformat (ctx=0x55555e756b40, reformat_type=MLX5DR_ACTION_TYP_REFORMAT_TNL_L3_TO_L2, num_of_hdrs=1 '\001', hdrs=0x7fffffffb720, log_bulk_size=1, flags=32) at ../drivers/net/mlx5/hws/mlx5dr_action.c:1594<o:p></o:p> #4 0x0000555557927826 in mlx5_tbl_multi_pattern_process (dev=0x555559167300 <rte_eth_devices>, tbl=0x17fdd1280, mpat=0x7fffffffb9c8, error=0x7fffffffd050) at ../drivers/net/mlx5/mlx5_flow_hw.c:4146<o:p></o:p> #5 0x000055555795f133 in mlx5_hw_build_template_table (dev=0x555559167300 <rte_eth_devices>, nb_action_templates=1 '\001', action_templates=0x555558dc1830 <port_template_info_pf+16>, at=0x7fffffffcf00, tbl=0x17fdd1280, error=0x7fffffffd050)<o:p></o:p> at ../drivers/net/mlx5/mlx5_flow_hw.c:4235<o:p></o:p> #6 0x00005555579022d3 in flow_hw_table_create (dev=0x555559167300 <rte_eth_devices>, table_cfg=0x7fffffffdec8, item_templates=0x555558dc1828 <port_template_info_pf+8>, nb_item_templates=1 '\001',<o:p></o:p> action_templates=0x555558dc1830 <port_template_info_pf+16>, nb_action_templates=1 '\001', error=0x7fffffffe0f8) at ../drivers/net/mlx5/mlx5_flow_hw.c:4401<o:p></o:p> #7 0x00005555577f89ec in flow_hw_template_table_create (dev=0x555559167300 <rte_eth_devices>, attr=0x5555589cc1e4 <table_attr_pf>, item_templates=0x555558dc1828 <port_template_info_pf+8>, nb_item_templates=1 '\001',<o:p></o:p> action_templates=0x555558dc1830 <port_template_info_pf+16>, nb_action_templates=1 '\001', error=0x7fffffffe0f8) at ../drivers/net/mlx5/mlx5_flow_hw.c:4589<o:p></o:p> #8 0x0000555556ca21e8 in mlx5_flow_table_create (dev=0x555559167300 <rte_eth_devices>, attr=0x5555589cc1e4 <table_attr_pf>, item_templates=0x555558dc1828 <port_template_info_pf+8>, nb_item_templates=1 '\001',<o:p></o:p> action_templates=0x555558dc1830 <port_template_info_pf+16>, nb_action_templates=1 '\001', error=0x7fffffffe0f8) at ../drivers/net/mlx5/mlx5_flow.c:9357<o:p></o:p> #9 0x0000555555c07c9a in rte_flow_template_table_create (port_id=0, table_attr=0x5555589cc1e4 <table_attr_pf>, pattern_templates=0x555558dc1828 <port_template_info_pf+8>, nb_pattern_templates=1 '\001',<o:p></o:p> actions_templates=0x555558dc1830 <port_template_info_pf+16>, nb_actions_templates=1 '\001', error=0x7fffffffe0f8) at ../lib/ethdev/rte_flow.c:1928<o:p></o:p> </Trace Excerpt><o:p></o:p> <o:p></o:p> Any comment or suggestions on this issue would be appreciated. Thanks in advance.<o:p></o:p> <o:p></o:p> Best regards,<o:p></o:p> Tao Li<o:p></o:p> <o:p></o:p> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="0" width="100%" align="center"> </div> From: Gregory Etelson <getelson@nvidia.com> Sent: 19 March 2024 14:25 To: Suanming Mou <suanmingm@nvidia.com>; Guvenc Gulce <guvenc.gulce@gmail.com>; users@dpdk.org <users@dpdk.org> Cc: Ori Kam <orika@nvidia.com>; Maayan Kashani <mkashani@nvidia.com> Subject: Re: mlx5: rte_flow template/async API raw_encap validation bug ?<o:p></o:p> <o:p></o:p> Hello Guvenc,<o:p></o:p> <o:p></o:p> Flow actions in MLX5 PMD actions template are translated according to these general rules:<o:p></o:p> <ol style="margin-top:0cm" start="1" type="1"> <li class="MsoNormal" style="color:black;margin-bottom:12.0pt;mso-list:l0 level1 lfo1"> If flow action configuration in template mask was not NULL, PMD constructs the action according to the action configuration parameters. PMD will use that pre-build action during async flow creation. The action parameters cannot be changed during async flow creation.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1">If flow action configuration in template mask was NULL, PMD ignores the action configuration in the template. The action will be constructed according to configuration data provided during async flow creation.<o:p></o:p></li></ol> <o:p></o:p> Before patch 2e543b6f18a2 ("net/mlx5: reuse reformat and modify actions in a table") the PMD ignored the RAW_ENCAP NULL mask configuration and used the action configuration for construction.<o:p></o:p> 2e543b6f18a2 does not allow access to RAW_ENCAP configuration if the action did not provide correct mask.<o:p></o:p> <o:p></o:p> If flow action configuration has several parameters, the action template can be partially translated - some action parameters will be provided with the template and other with async flow. In that case, if the action mask parameter has any non-zero value, it's configuration parameter will be used in a template. If the action mask parameter is 0, that parameter value will be provided during async flow.<o:p></o:p> <o:p></o:p> Partial action translation used for pre-defined flow actions. <o:p></o:p> <o:p></o:p> MLX5 PMD requires the `size` parameter of the RAW_ENCAP action during the template action translation. The action data can be provided ether with the template action configuration or with async flow.<o:p></o:p> Therefore, the RAW_ENCAP template configuration can be fully masked with the action size and data or partially masked with size only.<o:p></o:p> <o:p></o:p> Regards,<o:p></o:p> Gregory<o:p></o:p> <o:p></o:p> <o:p></o:p> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="0" width="100%" align="center"> </div> From: Suanming Mou <suanmingm@nvidia.com> Sent: Tuesday, March 19, 2024 02:24 To: Guvenc Gulce <guvenc.gulce@gmail.com>; users@dpdk.org <users@dpdk.org>; Gregory Etelson <getelson@nvidia.com> Cc: Ori Kam <orika@nvidia.com>; Maayan Kashani <mkashani@nvidia.com> Subject: RE: mlx5: rte_flow template/async API raw_encap validation bug ?<o:p></o:p> <o:p></o:p> Hi Guvenc,<o:p></o:p> <o:p></o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Guvenc Gulce <guvenc.gulce@gmail.com> Sent: Monday, March 18, 2024 6:26 PM To: users@dpdk.org Cc: Suanming Mou <suanmingm@nvidia.com>; Ori Kam <orika@nvidia.com> Subject: mlx5: rte_flow template/async API raw_encap validation bug ?<o:p></o:p> </div> <o:p></o:p> Hi all,<o:p></o:p> <o:p></o:p> It is great that we have rte_flow async/template api integrated to mlx5 <o:p></o:p> driver code and it is being established as the new standard rte_flow API.<o:p></o:p> <o:p></o:p> I have the following raw_encap problem when using the rte_flow async/template API <o:p></o:p> with mlx5 driver:<o:p></o:p> - raw_encap rte_flow action template fails during validation when the action mask<o:p></o:p> conf is NULL but this clearly contradicts the explanation from Suanming Mou's <o:p></o:p> commit 7f6daa490d9 which clearly states that the raw encap action mask is allowed <o:p></o:p> to be NULL.<o:p></o:p> <Excerpt from commit 7f6daa490d9><o:p></o:p> 2. RAW encap (encap_data: raw) <o:p></o:p> action conf (raw_data) <o:p></o:p> <o:p></o:p> a. action mask conf (not NULL) <o:p></o:p> - encap_data constant. <o:p></o:p> b. action mask conf (NULL) <o:p></o:p> - encap_data will change.<o:p></o:p> </Excerpt from commit 7f6daa490d9><o:p></o:p> <o:p></o:p> Commenting out the raw_encap validation would make it possible to create<o:p></o:p> rte_flow template with null mask conf which can be concretized later on. <o:p></o:p> Things seem to work after relaxing the rte_flow raw_encap validation.<o:p></o:p> The change would look like:<o:p></o:p> <o:p></o:p> [Suanming] I guess maybe it is due to the raw_encap and raw_decap combination. I added Gregory who added that code maybe can explain it better. <a id="OWAAMD01DCBAEF9F84ED9BA95BE5AFA046ABA" href="mailto:getelson@nvidia.com">@Gregory Etelson</a><o:p></o:p> <o:p></o:p> <Excerpt><o:p></o:p> diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c<o:p></o:p> index 35f1ed7a03..3f57fd9286 100644<o:p></o:p> --- a/drivers/net/mlx5/mlx5_flow_hw.c<o:p></o:p> +++ b/drivers/net/mlx5/mlx5_flow_hw.c<o:p></o:p> @@ -6020,10 +6020,10 @@ flow_hw_validate_action_raw_encap(const struct rte_flow_action *action,<o:p></o:p> const struct rte_flow_action_raw_encap *mask_conf = mask->conf;<o:p></o:p> const struct rte_flow_action_raw_encap *action_conf = action->conf;<o:p></o:p> <o:p></o:p> - if (!mask_conf || !mask_conf->size)<o:p></o:p> +/* if (!mask_conf || !mask_conf->size)<o:p></o:p> return rte_flow_error_set(error, EINVAL,<o:p></o:p> RTE_FLOW_ERROR_TYPE_ACTION, mask,<o:p></o:p> - "raw_encap: size must be masked");<o:p></o:p> + "raw_encap: size must be masked"); */<o:p></o:p> if (!action_conf || !action_conf->size)<o:p></o:p> return rte_flow_error_set(error, EINVAL,<o:p></o:p> RTE_FLOW_ERROR_TYPE_ACTION, action,<o:p></o:p> </Excerpt><o:p></o:p> <o:p></o:p> But this can not be the proper solution. Please advise a solution how to make the <o:p></o:p> raw_encap work with rte_flow template/async API. If relaxing the validation is ok, I can <o:p></o:p> also prepare and send a patch.<o:p></o:p> <o:p></o:p> Thanks in advance,<o:p></o:p> <o:p></o:p> Guvenc Gulce<o:p></o:p> <o:p></o:p> </div> </div> </div> </div> </body> </html>