810 VFIO , SRIOV, multi-process stats read DPDK testpmd.

[Bruce Richardson]

On Tue, Apr 29, 2025 at 08:04:04PM +0400, spyroot wrote:
>    Thank you very much, Bruce, for taking the time.
>    I'm observing odd behavior when PF either does not show stats for some
>    VF
>    or leaks VF stats from one to another.
>    i.e., generator A sent 100 packets to VF A and generator B 100 to VF B;
>    VF A and PF ICEN stats show 200 packets per VF stats.
>    (Note each flow is destined to the corresponding MAC of VF,
>    and prior to a generation, MAC entry is confirmed on the L2 switch, TX
>    VF.
>    So the packets aren't flooded, nor is it an unknown unicast frame).
>    The second condition, PF, shows zero stats for one VFs, but two RX
>    instances of testpmd
>    show correct statistics. (i.e A should see 100, B should see 100, A and
>    B
>    each report 100, but PF reports 0 for either A VF or B VF).
> 
>    The results are the same whether tested on the same node (k8s)
>    i.e lo-locate with TX PODs or on two different k8s nodes.
>    Condition two is observed if you do a single flow (i.e., hash on RX ide
>    land packets to the same RX queue). PF doesn't account for or report
>    correct values if you have two readers (i.e., two DPDK instances).
>    (Note: TX reports correct stats for A and B)
>    i.e. A sent 100, B sent 100
>    switch shows 200.
>    TX - A - VF A -- (L2 switch) -- (RX side) VF A -- testpmd
>    TX - B - VF B - - (L2 switch) -- (RX side) VF B -- testpmd.
>    On the TX side, 3 points of measurement.
>    VF TX stats, PF TX stats, L2 switch (total pkt,ppks etc) TX and
>    outgoing port RX.
>    Kind Regards,
>    Mus

Hi again,

do you see the same results/issue if you use a kernel driver for each VF
rather than a DPDK one? Issues with the stats reported by the PF are not
likely related to the specific driver or PMD using the VF, so I'm not sure
that you are seeing an issue with DPDK itself.

/Bruce

>    On Tue, Apr 29, 2025 at 5:22 PM Bruce Richardson
>    <[1]bruce.richardson at intel.com> wrote:
> 
>      On Mon, Apr 14, 2025 at 07:21:57PM +0400, spyroot wrote:
>      >    Hi Folks,
>      >
>      >    I'm observing some unexpected behavior related to how
>      statistics are
>      >    retrieved from a Physical Function (PF) on an Intel 810 NIC.
>      >
>      >    Scenario: I have two dpdk-testpmd instances running in separate
>      >    Kubernetes pods (same worker node). Each instance uses the -a
>      flag to
>      >    bind to a different VF. (i.e to have consistent port id 0)
>      >
>      >    Questions:
>      >     1. PF Statistics and 64B Line Rate:
>      >        I'm noticing that the RX packet-per-second value reported
>      on the PF
>      >        side for a given VF is higher than the theoretical maximum
>      for
>      >        64-byte packets.
>      >           + Does the Intel 810 PMD apply any kind of optimization,
>      >             offloading, or fast path processing when two VFs
>      (e.g., A and
>      >             B) are on the same PF?
>      This wouldn't be something that the PMD does. The forwarding from VF
>      to VF,
>      or PF to VF would happen internally in the hardware.
>      >     2. Concurrent Stats Polling:
>      >           + When two separate dpdk-testpmd processes are running
>      (in pod A
>      >             and pod B), does the PMD or driver layer support
>      concurrent
>      >             reading of PF statistics?
>      Looking at the iavf PMD, the reading of stats is done by sending an
>      adminq
>      message to the PF and reading the response. Any serialization of
>      stats
>      reading would then be done at the PF or adminq management level. The
>      VF
>      should not need to worry about whether another VF is reading the
>      stats at
>      the same time. [In fact it would be a serious bug if one VF needed
>      to be
>      aware of what other VFs were doing, since different VFs could be
>      attached
>      to different virtual machines which should be isolated from each
>      other]
>      >           + Is there any locking or synchronization mechanism
>      involved
>      >             when multiple testpmd instances attempt to pull stats
>      from the
>      >             same PF simultaneously? ( in essence, does a
>      firmware/OF
>      >             support concurrent read).
>      >
>      See above.
>      /Bruce
>      >    Thank you,
>      > cd /usr/local/bin && dpdk-testpmd \
>      >   --main-lcore \$main -l \$cores -n 4 \
>      >   --socket-mem 2048 \
>      >   --proc-type auto --file-prefix testpmd_rx0 \
>      >   -a \$PCIDEVICE_INTEL_COM_DPDK \
>      >   -- --forward-mode=rxonly --auto-start --stats-period 1'"
>      >
>      > cd /usr/local/bin && dpdk-testpmd \
>      >   --main-lcore \$main -l \$cores -n 4 \
>      >   --socket-mem 2048 \
>      >   --proc-type auto --file-prefix testpmd_rx1 \
>      >   -a \$PCIDEVICE_INTEL_COM_DPDK \
>      >   -- --forward-mode=rxonly --auto-start --stats-period 1'"
> 
> References
> 
>    1. mailto:bruce.richardson at intel.com