<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hi all,<o:p></o:p> I hit the exact same problem reported by Yan.<o:p></o:p> I’m using:<o:p></o:p> <ul style="margin-top:0in" type="disc"> <li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">2 Mellanox CX5 MT28800 installed on 2 different servers, connected together <o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Device FW (as reported by DPDK): 16.31.1014<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">DPDK 19.11.5 (from 6WindGate actually)<o:p></o:p></li></ul> <o:p> </o:p> I sent roughly 360M packets from one server to the other using “testpmd” (in –forward-mode=txonly).<o:p></o:p> My DPDK application on the other server is reporting the following xstats counters (:<o:p></o:p> <o:p> </o:p> CounterName PORT0 PORT1 TOTAL<o:p></o:p> rx_good_packets: 76727920, 0, 76727920<o:p></o:p> tx_good_packets: 0, 0, 0<o:p></o:p> rx_good_bytes: 4910586880, 0, 4910586880<o:p></o:p> tx_good_bytes: 0, 0, 0<o:p></o:p> rx_missed_errors: 0, 0, 0<o:p></o:p> rx_errors: 0, 0, 0<o:p></o:p> tx_errors: 0, 0, 0<o:p></o:p> rx_mbuf_allocation_errors: 0, 0, 0<o:p></o:p> rx_q0packets: 0, 0, 0<o:p></o:p> rx_q0bytes: 0, 0, 0<o:p></o:p> rx_q0errors: 0, 0, 0<o:p></o:p> rx_q1packets: 0, 0, 0<o:p></o:p> rx_q1bytes: 0, 0, 0<o:p></o:p> rx_q1errors: 0, 0, 0<o:p></o:p> rx_q2packets: 0, 0, 0<o:p></o:p> rx_q2bytes: 0, 0, 0<o:p></o:p> rx_q2errors: 0, 0, 0<o:p></o:p> rx_q3packets: 0, 0, 0<o:p></o:p> rx_q3bytes: 0, 0, 0<o:p></o:p> rx_q3errors: 0, 0, 0<o:p></o:p> rx_q4packets: 0, 0, 0<o:p></o:p> rx_q4bytes: 0, 0, 0<o:p></o:p> rx_q4errors: 0, 0, 0<o:p></o:p> rx_q5packets: 76727920, 0, 76727920<o:p></o:p> rx_q5bytes: 4910586880, 0, 4910586880<o:p></o:p> rx_q5errors: 0, 0, 0<o:p></o:p> rx_q6packets: 0, 0, 0<o:p></o:p> rx_q6bytes: 0, 0, 0<o:p></o:p> rx_q6errors: 0, 0, 0<o:p></o:p> rx_q7packets: 0, 0, 0<o:p></o:p> rx_q7bytes: 0, 0, 0<o:p></o:p> rx_q7errors: 0, 0, 0<o:p></o:p> rx_q8packets: 0, 0, 0<o:p></o:p> rx_q8bytes: 0, 0, 0<o:p></o:p> rx_q8errors: 0, 0, 0<o:p></o:p> rx_q9packets: 0, 0, 0<o:p></o:p> rx_q9bytes: 0, 0, 0<o:p></o:p> rx_q9errors: 0, 0, 0<o:p></o:p> rx_q10packets: 0, 0, 0<o:p></o:p> rx_q10bytes: 0, 0, 0<o:p></o:p> rx_q10errors: 0, 0, 0<o:p></o:p> rx_q11packets: 0, 0, 0<o:p></o:p> rx_q11bytes: 0, 0, 0<o:p></o:p> rx_q11errors: 0, 0, 0<o:p></o:p> tx_q0packets: 0, 0, 0<o:p></o:p> tx_q0bytes: 0, 0, 0<o:p></o:p> rx_wqe_err: 0, 0, 0<o:p></o:p> rx_port_unicast_packets: 360316064, 0, 360316064<o:p></o:p> rx_port_unicast_bytes: 23060228096, 0, 23060228096<o:p></o:p> tx_port_unicast_packets: 0, 0, 0<o:p></o:p> tx_port_unicast_bytes: 0, 0, 0<o:p></o:p> rx_port_multicast_packets: 0, 0, 0<o:p></o:p> rx_port_multicast_bytes: 0, 0, 0<o:p></o:p> tx_port_multicast_packets: 0, 0, 0<o:p></o:p> tx_port_multicast_bytes: 0, 0, 0<o:p></o:p> rx_port_broadcast_packets: 0, 0, 0<o:p></o:p> rx_port_broadcast_bytes: 0, 0, 0<o:p></o:p> tx_port_broadcast_packets: 0, 0, 0<o:p></o:p> tx_port_broadcast_bytes: 0, 0, 0<o:p></o:p> tx_packets_phy: 0, 0, 0<o:p></o:p> rx_packets_phy: 0, 0, 0<o:p></o:p> rx_crc_errors_phy: 0, 0, 0<o:p></o:p> tx_bytes_phy: 0, 0, 0<o:p></o:p> rx_bytes_phy: 0, 0, 0<o:p></o:p> rx_in_range_len_errors_phy 0, 0, 0<o:p></o:p> rx_symbol_err_phy: 0, 0, 0<o:p></o:p> rx_discards_phy: 0, 0, 0<o:p></o:p> tx_discards_phy: 0, 0, 0<o:p></o:p> tx_errors_phy: 0, 0, 0<o:p></o:p> rx_out_of_buffer: 0, 0, 0<o:p></o:p> <o:p> </o:p> So rx_good_packets is roughly 76M pkts, while rx_port_unicast_packets has counted correctly all 360M pkts sent by testpmd.<o:p></o:p> Of course my application layer has been able to dequeue from the DPDK port only 76M pkts so the remaining (rx_port_unicast_packets- rx_good_packets) got lost but are not reported in the “imissed” counter or “ierrors” counter of rte_eth_stats…<o:p></o:p> <o:p> </o:p> It would be not so easy for me to test against latest DPDK… also from what Yan has reported, the issue is still there in DPDK stable 20.11.3…<o:p></o:p> <o:p> </o:p> @Mellanox maintainers: any update on this issue? Is there a workaround to get the dropped packet back into the rte_eth_stats counters?<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Thanks <o:p></o:p> <o:p> </o:p> Francesco Montorsi<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com> Sent: Friday, October 29, 2021 2:44 AM To: Gerry Wan <gerryw@stanford.edu>; Asaf Penso <asafp@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad <matan@nvidia.com>; Raslan Darawsheh <rasland@nvidia.com> Cc: Martin Weiser <martin.weiser@allegro-packets.com>; David Marchand <david.marchand@redhat.com>; users@dpdk.org Subject: RE: mlx5 VF packet lost between rx_port_unicast_packets and rx_good_packets<o:p></o:p> </div> </div> <o:p> </o:p> Hi,<o:p></o:p> <o:p> </o:p> Yes, it’s always zero…<o:p></o:p> It seems dpdk-stable-20.11-3 already include this patch.<o:p></o:p> [xiaopiya@fedora30 dpdk-stable-20.11.3]$ patch -p1 < ./4-4-net-mlx5-fix-imissed-statistics.diff<o:p></o:p> patching file drivers/net/mlx5/linux/mlx5_os.c<o:p></o:p> Reversed (or previously applied) patch detected! Assume -R? [n] n<o:p></o:p> Apply anyway? [n] ^C<o:p></o:p> [xiaopiya@fedora30 dpdk-stable-20.11.3]$ grep -r "mlx5_queue_counter_id_prepare" ./<o:p></o:p> ./drivers/net/mlx5/linux/mlx5_os.c:mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)<o:p></o:p> ./drivers/net/mlx5/linux/mlx5_os.c: mlx5_queue_counter_id_prepare(eth_dev);<o:p></o:p> ./drivers/net/mlx5/linux/mlx5_os.c.rej:+mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)<o:p></o:p> ./drivers/net/mlx5/linux/mlx5_os.c.rej:+ mlx5_queue_counter_id_prepare(eth_dev);<o:p></o:p> ./4-4-net-mlx5-fix-imissed-statistics.diff:+mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)<o:p></o:p> ./4-4-net-mlx5-fix-imissed-statistics.diff:+ mlx5_queue_counter_id_prepare(eth_dev);<o:p></o:p> <o:p> </o:p> Thank you.<o:p></o:p> <o:p> </o:p> Best regards<o:p></o:p> Yan Xiaoping<o:p></o:p> <o:p> </o:p> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: Gerry Wan <<a href="mailto:gerryw@stanford.edu">gerryw@stanford.edu</a>> Sent: 2021年10月28日 9:58 To: Yan, Xiaoping (NSB - CN/Hangzhou) <<a href="mailto:xiaoping.yan@nokia-sbell.com">xiaoping.yan@nokia-sbell.com</a>> Cc: Martin Weiser <<a href="mailto:martin.weiser@allegro-packets.com">martin.weiser@allegro-packets.com</a>>; David Marchand <<a href="mailto:david.marchand@redhat.com">david.marchand@redhat.com</a>>; Asaf Penso <<a href="mailto:asafp@nvidia.com">asafp@nvidia.com</a>>; Slava Ovsiienko <<a href="mailto:viacheslavo@nvidia.com">viacheslavo@nvidia.com</a>>; Matan Azrad <<a href="mailto:matan@nvidia.com">matan@nvidia.com</a>>; Raslan Darawsheh <<a href="mailto:rasland@nvidia.com">rasland@nvidia.com</a>>; <a href="mailto:users@dpdk.org">users@dpdk.org</a> Subject: Re: mlx5 VF packet lost between rx_port_unicast_packets and rx_good_packets<o:p></o:p> </div> <o:p> </o:p> <div> <div> Are the rx_missed_errors/rx_out_of_buffer counters always showing 0 no matter how fast you push your generator?<o:p></o:p> </div> <div> <o:p> </o:p> </div> I had a similar issue with missing counters on DPDK 20.11 that was fixed on 21.05 by applying this patch:<o:p></o:p> <div> <a href="http://patchwork.dpdk.org/project/dpdk/patch/1614249901-307665-5-git-send-email-matan@nvidia.com/">http://patchwork.dpdk.org/project/dpdk/patch/1614249901-307665-5-git-send-email-matan@nvidia.com/</a><o:p></o:p> <div> <o:p> </o:p> </div> <div> Potentially relevant thread:<o:p></o:p> </div> </div> <div> <a href="https://inbox.dpdk.org/users/CAAcwi38rs2Vk9MKhRGS3kAK+=dYAnDdECT7f+Ts-f13cANYB+Q@mail.gmail.com/">https://inbox.dpdk.org/users/CAAcwi38rs2Vk9MKhRGS3kAK+=dYAnDdECT7f+Ts-f13cANYB+Q@mail.gmail.com/</a><o:p></o:p> </div> </div> <o:p> </o:p> <div> <div> On Wed, Oct 27, 2021 at 6:39 PM Yan, Xiaoping (NSB - CN/Hangzhou) <<a href="mailto:xiaoping.yan@nokia-sbell.com">xiaoping.yan@nokia-sbell.com</a>> wrote:<o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> Hi, I checked the counter from PF with ethtool -S, there is no counter named 'rx_prio0_buf_discard' Anyway, I checked all counters from ethtool output, there is not any counter reflects the dropped packets. Any suggestion from mlx maintainer? @Matan Azrad @Asaf Penso @Slava Ovsiienko @Raslan Darawsheh Thank you. Best regards Yan Xiaoping -----Original Message----- From: Martin Weiser <<a href="mailto:martin.weiser@allegro-packets.com" target="_blank">martin.weiser@allegro-packets.com</a>> Sent: 2021年10月27日 15:54 To: Yan, Xiaoping (NSB - CN/Hangzhou) <<a href="mailto:xiaoping.yan@nokia-sbell.com" target="_blank">xiaoping.yan@nokia-sbell.com</a>>; David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>> Cc: Asaf Penso <<a href="mailto:asafp@nvidia.com" target="_blank">asafp@nvidia.com</a>>; <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a>; Slava Ovsiienko <<a href="mailto:viacheslavo@nvidia.com" target="_blank">viacheslavo@nvidia.com</a>>; Matan Azrad <<a href="mailto:matan@nvidia.com" target="_blank">matan@nvidia.com</a>>; Raslan Darawsheh <<a href="mailto:rasland@nvidia.com" target="_blank">rasland@nvidia.com</a>> Subject: Re: mlx5 VF packet lost between rx_port_unicast_packets and rx_good_packets Hi, you may want to check the counter 'rx_prio0_buf_discard' with ethtool (which is not available in DPDK xstats as it seems that this counter is global for the card and not available per port). I opened a ticket a while ago regarding this issue: <a href="https://bugs.dpdk.org/show_bug.cgi?id=749" target="_blank">https://bugs.dpdk.org/show_bug.cgi?id=749</a> Best regards, Martin Am 27.10.21 um 08:17 schrieb Yan, Xiaoping (NSB - CN/Hangzhou): > Hi, > > I tried with dpdk 20.11-3 downloaded from > <a href="https://fast.dpdk.org/rel/dpdk-20.11.3.tar.xz" target="_blank">https://fast.dpdk.org/rel/dpdk-20.11.3.tar.xz</a> > Problem still exist: > 1. there is packet loss with 2mpps (small packet), 2. no counter for > the dropped packet in NIC. > > traffic generator stats: sends 41990938, receives back 41986105, lost > 4833 testpmd fwd stats: RX-packets: 41986110, TX-packets: 41986110 > port xstats: rx_unicast_packets: 41990938 (all packets reached to the NIC port), rx_good_packets: 41986111 (some is lost), but there is not any counter of the lost packet. > > Here is the log: > [root@up-0 /]# dpdk-testpmd -l "2,3" --legacy-mem --socket-mem > "5000,0" -a 0000:03:06.7 -- -i --nb-cores=1 --portmask=0x1 --rxd=512 > --txd=512 > EAL: Detected 28 lcore(s) > EAL: Detected 2 NUMA nodes > EAL: Detected static linkage of DPDK > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'VA' > EAL: No available hugepages reported in hugepages-2048kB > EAL: Probing VFIO support... > EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:03:06.7 > (socket 0) > mlx5_pci: cannot bind mlx5 socket: Read-only file system > mlx5_pci: Cannot initialize socket: Read-only file system > EAL: No legacy callbacks, legacy socket not created Interactive-mode > selected > testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, > socket=0 > testpmd: preferred mempool ops selected: ring_mp_mc > > Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. > > Configuring Port 0 (socket 0) > Port 0: 7A:9A:8A:A6:86:93 > Checking link statuses... > Done > testpmd> port stop 0 > Stopping ports... > Checking link statuses... > Done > testpmd> vlan set filter on 0 > testpmd> rx_vlan add 767 0 > testpmd> port start 0 > Port 0: 7A:9A:8A:A6:86:93 > Checking link statuses... > Done > testpmd> set fwd 5tswap > Set 5tswap packet forwarding mode > testpmd> start > 5tswap packet forwarding - ports=1 - cores=1 - streams=1 - NUMA > support enabled, MP allocation mode: native Logical Core 3 (socket 0) forwards packets on 1 streams: > RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) > peer=02:00:00:00:00:00 > > 5tswap packet forwarding packets/burst=32 > nb forwarding cores=1 - nb forwarding ports=1 > port 0: RX queue number: 1 Tx queue number: 1 > Rx offloads=0x200 Tx offloads=0x0 > RX queue: 0 > RX desc=512 - RX free threshold=64 > RX threshold registers: pthresh=0 hthresh=0 wthresh=0 > RX Offloads=0x200 > TX queue: 0 > TX desc=512 - TX free threshold=0 > TX threshold registers: pthresh=0 hthresh=0 wthresh=0 > TX offloads=0x0 - TX RS bit threshold=0 > > testpmd> show fwd stats all > > ---------------------- Forward statistics for port 0 ---------------------- > RX-packets: 41986110 RX-dropped: 0 RX-total: 41986110 > TX-packets: 41986110 TX-dropped: 0 TX-total: 41986110 > > ---------------------------------------------------------------------- > ------ > > testpmd> show port xstats 0 > ###### NIC extended statistics for port 0 > rx_good_packets: 41986111 > tx_good_packets: 41986111 > rx_good_bytes: 3106973594 > tx_good_bytes: 3106973594 > rx_missed_errors: 0 > rx_errors: 0 > tx_errors: 0 > rx_mbuf_allocation_errors: 0 > rx_q0_packets: 41986111 > rx_q0_bytes: 3106973594 > rx_q0_errors: 0 > tx_q0_packets: 41986111 > tx_q0_bytes: 3106973594 > rx_wqe_errors: 0 > rx_unicast_packets: 41990938 > rx_unicast_bytes: 3107329412 > tx_unicast_packets: 41986111 > tx_unicast_bytes: 3106973594 > rx_multicast_packets: 1 > rx_multicast_bytes: 114 > tx_multicast_packets: 0 > tx_multicast_bytes: 0 > rx_broadcast_packets: 5 > rx_broadcast_bytes: 1710 > tx_broadcast_packets: 0 > tx_broadcast_bytes: 0 > tx_phy_packets: 0 > rx_phy_packets: 0 > rx_phy_crc_errors: 0 > tx_phy_bytes: 0 > rx_phy_bytes: 0 > rx_phy_in_range_len_errors: 0 > rx_phy_symbol_errors: 0 > rx_phy_discard_packets: 0 > tx_phy_discard_packets: 0 > tx_phy_errors: 0 > rx_out_of_buffer: 0 > tx_pp_missed_interrupt_errors: 0 > tx_pp_rearm_queue_errors: 0 > tx_pp_clock_queue_errors: 0 > tx_pp_timestamp_past_errors: 0 > tx_pp_timestamp_future_errors: 0 > tx_pp_jitter: 0 > tx_pp_wander: 0 > tx_pp_sync_lost: 0 > testpmd> q > Command not found > testpmd> exit > Command not found > testpmd> quit > Telling cores to stop... > Waiting for lcores to finish... > > ---------------------- Forward statistics for port 0 ---------------------- > RX-packets: 41986112 RX-dropped: 0 RX-total: 41986112 > TX-packets: 41986112 TX-dropped: 0 TX-total: 41986112 > > ---------------------------------------------------------------------- > ------ > > Best regards > Yan Xiaoping > > -----Original Message----- > From: David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>> > Sent: 2021年10月18日 18:45 > To: Yan, Xiaoping (NSB - CN/Hangzhou) <<a href="mailto:xiaoping.yan@nokia-sbell.com" target="_blank">xiaoping.yan@nokia-sbell.com</a>> > Cc: Asaf Penso <<a href="mailto:asafp@nvidia.com" target="_blank">asafp@nvidia.com</a>>; <a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a>; Slava Ovsiienko > <<a href="mailto:viacheslavo@nvidia.com" target="_blank">viacheslavo@nvidia.com</a>>; Matan Azrad <<a href="mailto:matan@nvidia.com" target="_blank">matan@nvidia.com</a>>; Raslan > Darawsheh <<a href="mailto:rasland@nvidia.com" target="_blank">rasland@nvidia.com</a>> > Subject: Re: mlx5 VF packet lost between rx_port_unicast_packets and > rx_good_packets > > On Mon, Oct 18, 2021 at 11:28 AM Yan, Xiaoping (NSB - CN/Hangzhou) <<a href="mailto:xiaoping.yan@nokia-sbell.com" target="_blank">xiaoping.yan@nokia-sbell.com</a>> wrote: >> I have cloned dpdk code from github >> >> [xiaopiya@fedora30 dpdk]$ git remote -v origin >> <a href="https://github.com/DPDK/dpdk.git" target="_blank">https://github.com/DPDK/dpdk.git</a> (fetch) origin >> <a href="https://github.com/DPDK/dpdk.git" target="_blank">https://github.com/DPDK/dpdk.git</a> (push) >> >> which tag should I use? >> >> Or do I have to download 20.11.3 from <a href="http://git.dpdk.org" target="_blank"> git.dpdk.org</a>? >> >> Sorry, I don’t know the relation between <a href="https://github.com/DPDK" target="_blank"> https://github.com/DPDK</a> and <a href="http://git.dpdk.org" target="_blank">git.dpdk.org</a>? > Github DPDK/dpdk repo is a replication of the main repo hosted on <a href="http://dpdk.org" target="_blank"> dpdk.org</a> servers. > > The official git repos and releases tarballs are on <a href="http://dpdk.org" target="_blank"> dpdk.org</a> servers. > The list of official releases tarballs is at: > <a href="http://core.dpdk.org/download/" target="_blank">http://core.dpdk.org/download/</a> The main repo git is at: > <a href="https://git.dpdk.org/dpdk/" target="_blank">https://git.dpdk.org/dpdk/</a> The LTS/stable releases repo git is at: > <a href="https://git.dpdk.org/dpdk-stable/" target="_blank">https://git.dpdk.org/dpdk-stable/</a> > > > -- > David Marchand ><o:p></o:p> </blockquote> </div> </div> </body> </html>