<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="en-DE" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> <pre>Hi all, Hi Dariusz,<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre>Thanks a lot for your help so far. We really appreciate it.<o:p></o:p></pre> <pre>I just want to touch base with this question which was asked by my colleague Tao a while back.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Our question is actually quite simple. Issuing the commands listed below on a ConnectX-6 Dx Card breaks the <o:p></o:p></pre> <pre>bifurcated nature of the mlx5 driver in linux kernel for PF1. (No traffic is forwarded to linux kernel anymore on PF1)<o:p></o:p></pre> <pre>You don’t need to start any testpmd or dpdk application. Just issuing the following commands below breaks the PF1 <o:p></o:p></pre> <pre>in linux kernel already.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>sudo devlink dev eswitch set pci/0000:8a:00.0 mode switchdev<o:p></o:p></pre> <pre>sudo devlink dev eswitch set pci/0000:8a:00.1 mode switchdev<o:p></o:p></pre> <pre>sudo devlink dev param set pci/0000:8a:00.0 name esw_multiport value true cmode runtime<o:p></o:p></pre> <pre>sudo devlink dev param set pci/0000:8a:00.1 name esw_multiport value true cmode runtime<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre>----<test environment>-----<o:p></o:p></pre> <pre>pci/0000:8a:00.0:<o:p></o:p></pre> <pre> driver mlx5_core<o:p></o:p></pre> <pre> versions:<o:p></o:p></pre> <pre> fixed:<o:p></o:p></pre> <pre> fw.psid MT_0000000359<o:p></o:p></pre> <pre> running:<o:p></o:p></pre> <pre> fw.version 22.39.2048<o:p></o:p></pre> <pre> fw 22.39.2048<o:p></o:p></pre> <pre>Linux kernel version: 6.6.16<o:p></o:p></pre> <pre>DPDK: 23.11 (But not really needed to reproduce the issue)<o:p></o:p></pre> <pre>----</test environment>------<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre>This makes the eswitch multiport feature for us unusable. Could you please advise whether we are missing smt here ?<o:p></o:p></pre> <pre>As we are really keen to use this feature.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Thanks & Regards<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre>Guvenc Gulce<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre>-------------------- previous email exchange -------------------------------<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Hi Dariusz,<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>It is very appreciated that you took a look at the issue and provided suggestions. This time, we again performed tests using two directly connected machines and focused on ICMP (IPv4) packets in addition to ICMPv6 packets mentioned in the original problem description. The issue remains the same. I would like to highlight two points in our setup:<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre> 1. ICMP packets immediately cannot be captured on PF1 right after setting the nic into the multiport eswitch mode. And if I switch off the multiport eswitch mode by using following two commands, ICMP communication is resumed immediately, which shall prove that configs, such as firewall, on the system are correct. I would also assume it has little to do with a running DPDK application, as communication is already broken before starting an application like testpmd.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>sudo devlink dev param set pci/0000:3b:00.0 name esw_multiport value false cmode runtime<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>sudo devlink dev param set pci/0000:3b:00.1 name esw_multiport value false cmode runtime<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre> 1. In this setup, we do not use MLNX_OFED drivers but rely on the upstream Mellanox drivers from Linux kernel 6.5.0 (which is greater than the suggested kernel version 6.3). Would that make a difference? Could you share some more detailed information regarding the environment setup on your side? The firmware version we are using for Mellanox ConnectX-6 is 22.39.1002.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Looking forward to your further reply. Thanks in advance.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Best regards,<o:p></o:p></pre> <pre>Tao Li<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>From: Dariusz Sosnowski <<a href="https://mails.dpdk.org/listinfo/users">dsosnowski at nvidia.com</a>><o:p></o:p></pre> <pre>Date: Friday, 19. April 2024 at 19:30<o:p></o:p></pre> <pre>To: Tao Li <<a href="https://mails.dpdk.org/listinfo/users">byteocean at hotmail.com</a>>, <a href="https://mails.dpdk.org/listinfo/users">users at dpdk.org</a> <<a href="https://mails.dpdk.org/listinfo/users">users at dpdk.org</a>><o:p></o:p></pre> <pre>Cc: <a href="https://mails.dpdk.org/listinfo/users">tao.li06 at sap.com</a> <<a href="https://mails.dpdk.org/listinfo/users">tao.li06 at sap.com</a>><o:p></o:p></pre> <pre>Subject: RE: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver)<o:p></o:p></pre> <pre>Hi,<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>I could not reproduce the issue locally with testpmd, with flow isolation enabled. I can see ICMP packets passing both ways to kernel interfaces of PF0 and PF1.<o:p></o:p></pre> <pre>Without flow isolation, it is expected that traffic coming to the host will be hijacked by DPDK (depending on the MAC address, multicast config and promiscuous mode).<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Could you please run testpmd with the following command line parameters and execute the following commands?<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Testpmd command line:<o:p></o:p></pre> <pre> dpdk-testpmd -a 3b:00.0,dv_flow_en=2,representor=pf0-1vf0 -- --flow-isolate-all -i<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Testpmd commands:<o:p></o:p></pre> <pre> port stop all<o:p></o:p></pre> <pre> flow configure 0 queues_number 4 queues_size 64<o:p></o:p></pre> <pre> flow configure 1 queues_number 4 queues_size 64<o:p></o:p></pre> <pre> flow configure 2 queues_number 4 queues_size 64<o:p></o:p></pre> <pre> flow configure 3 queues_number 4 queues_size 64<o:p></o:p></pre> <pre> port start 0<o:p></o:p></pre> <pre> port start 1<o:p></o:p></pre> <pre> port start 2<o:p></o:p></pre> <pre> port start 3<o:p></o:p></pre> <pre> set verbose 1<o:p></o:p></pre> <pre> set fwd rxonly<o:p></o:p></pre> <pre> start<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>With this testpmd running, could you please test if both PF0 and PF1 kernel interfaces are reachable and all packets pass?<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Best regards,<o:p></o:p></pre> <pre>Dariusz Sosnowski<o:p></o:p></pre> <o:p> </o:p> </div> </body> </html>