[dpdk-dev] [PATCH v11 0/7] Expanded statistics reporting

Remy Horton remy.horton at intel.com
Thu Mar 9 17:25:27 CET 2017

This patchset consists of three libraries: A Metrics library for
distributing device information, a library that calculates bit-rate
statistics, and a library that calculates latency statistics. The
latter two libraries make use of the first library.

Metrics Library
The Metrics library implements a mechanism by which producers can
publish numeric information for later querying by consumers.
In practice producers will typically be other libraries or
primary processes, whereas consumers will typically be applications.

Metrics themselves are statistics that are not generated by PMDs.
Metric information is populated using a push model, where producers
update the values contained within the metric library by calling an
update function on the relevant metrics. Consumers receive metric
information by querying the central metric data, which is held
in shared memory so it can also be accessed by secondary processes.

For each metric, a separate value is maintained for each port id,
and when publishing metric values the producers need to specify
which port is being updated. In addition there is a special id
RTE_METRICS_GLOBAL that is intended for global statistics that are
not associated with any individual device. Since the metrics
library is self-contained, the only restriction on port numbers is
that they are less than RTE_MAX_ETHPORTS - there is no requirement
for the ports to actually exist.

Metrics must first be registered, which is the way producers declare
the names of the metrics they will be publishing. Registration can
either be done individually, or as a group as a metric set. The
intention is for all metrics in a set to be updated in one go,
although it is also possible for metrics within a set to be updated
individually. It is up to the producers to update metrics as required.

Bit-rate statistics library
The bit-rate library calculates the mean, the exponentially-weighted
moving average, and peak bit-rates for each active port (i.e. network
device). These statistics are then reported via the metrics library
using the following names: mean_bits_in, mean_bits_out, ewma_bits_in,
ewma_bits_out, peak_bits_in, and peak_bits_out. The sampling window 
used for calculation is decided by the application requiring the statistics.

Latency statistics library
The latency statistics library calculates the port-to-port latency of
packet processing by a DPDK application, reporting the minimum, average,
and maximum nano-seconds that packet processing takes, as well as the
jitter in processing delay. These statistics are then reported via the
metrics library using the following names: min_latency_ns, avg_latency_ns,
mac_latency_ns, and jitter_ns.

For more details on the metrics library and the Bit-rate and Latency
components see the Programmer's Guide updates in the patchset.
v11 changes:
* Rebased
* .map references to 17.02 changed to 17.05
* Release ntoes moved to release_17_05.rst
* Bit-rate library now also gives unfiltered average

v10 changes:
* Rebased
* Relocated some config-related directives.
* Removed incorrect capitalisations in API docs.
* Formatting & detail corrections in release notes.
* Moved around struct member descriptions.
* Rewritten rte_metrics.h file description.
* Rewritten description of RTE_METRICS_GLOBAL.
* Used 'producers' and 'consumers' as terms.
* Removed markup (bold text) in Doxygen tags.
* Added programming guide section.

v9 changes:
* Updated .map files to reflect function changes in v8
* Fixed rte_malloc() of zero bytes in proc_info when no metrics exist
* Fixed rte_metrics_init not being called explicitly in testpmd

v8 changes:
* Release notes correction
* Updated copyright years
* rte_metric_init() takes socket id & must be explicitly called
* rte_metrics_reg_metric renamed to rte_metrics_reg_name()
* rte_metrics_update_metric() renamed to rte_metrics_update_value()
* Doxygen updates
* Changed malloc()/free() to rte_malloc()/rte_free()
* Removed redundant memset()
* rte_stats_bitrates_s renamed to rte_stats_bitrates_s
* Split mbuf change to own patch for visibility
* CYCLES_PER_NS now a static inline function
* latency: "hidden" pthread creation now has polling API instead.
* Struct declarations and variable definitions cleaned up
* Double initialization of the latency library now returns -EEXIST
* MAINTAINERS entry for layenctstats in correct section

v7 changes:
* Multiple changes to rte_metrics.h doxygen documentation
* Split apart latency patch into lib, test-pmd, & proc_info parts
* Reordered patches by functionality
* Insufficent capacity return value changed from -ERANGE to actual size
* Cache alignment in bitrate library
* Tightened up const usage to avoid STATIC_CONST_CHAR_ARRAY warning
* Reshma reinstated as author for (now split) latency patch
* Rebase to master

v6 changes:
* Metrics display now has "Non port specific" rather than "port -1"
* Fixed sign issue in EWMA delta calculation
* Rebased to master

v5 changes:
* Updated Shared Library Versions in release notes
* Merged in Reshma's latencystats library

v4 changes:
* References to 16.11 changed to 17.02
* Fetching of non-port values was broken
* Added sanity checks to value fetching
* rte_stat_value renamed to rte_metric_value
* Corrected doxygen descriptions
* Added MAINTAINERS entries
* Added #ifdef directives to bitrate code in test-pmd

v3 changes:
* Marked rte_stats_bitrate_s as internal
* Minor integer roundoff correction
* Coding style corrections
* Removed spurious object allocation
* Changes to rte_metrics.[ch] moved from Patch 2/3 to 1/3.
* Reintroduced non-port values (RTE_METRICS_NONPORT)
* Added spinlocks to metric library
* Removed spurious test registration/update
* Added release notes entries

v2 changes:
* Uses a new metrics library rather than being part of ethdev

Harry van Haaren (3):
  mbuf: add a timestamp to the mbuf for latencystats
  lib: added new library for latency stats
  app/test-pmd: add latency statistics calculation

Remy Horton (3):
  lib: add information metrics library
  lib: add bitrate statistics library
  app/test-pmd: add bitrate statistics calculation

Reshma Pattan (1):
  app/proc_info: add metrics displaying

 MAINTAINERS                                        |  12 +
 app/proc_info/main.c                               |  79 ++++-
 app/test-pmd/parameters.c                          |  20 +-
 app/test-pmd/testpmd.c                             |  78 ++++-
 app/test-pmd/testpmd.h                             |   6 +-
 config/common_base                                 |  15 +
 doc/api/doxy-api-index.md                          |   3 +
 doc/api/doxy-api.conf                              |   3 +
 doc/guides/prog_guide/index.rst                    |   1 +
 doc/guides/prog_guide/metrics_lib.rst              | 299 +++++++++++++++++
 doc/guides/rel_notes/release_17_02.rst             |   3 +
 doc/guides/rel_notes/release_17_05.rst             |  18 +
 lib/Makefile                                       |   3 +
 lib/librte_bitratestats/Makefile                   |  53 +++
 lib/librte_bitratestats/rte_bitrate.c              | 141 ++++++++
 lib/librte_bitratestats/rte_bitrate.h              |  80 +++++
 .../rte_bitratestats_version.map                   |   9 +
 lib/librte_latencystats/Makefile                   |  56 ++++
 lib/librte_latencystats/rte_latencystats.c         | 366 +++++++++++++++++++++
 lib/librte_latencystats/rte_latencystats.h         | 154 +++++++++
 .../rte_latencystats_version.map                   |  11 +
 lib/librte_mbuf/rte_mbuf.h                         |   3 +
 lib/librte_metrics/Makefile                        |  51 +++
 lib/librte_metrics/rte_metrics.c                   | 299 +++++++++++++++++
 lib/librte_metrics/rte_metrics.h                   | 240 ++++++++++++++
 lib/librte_metrics/rte_metrics_version.map         |  13 +
 mk/rte.app.mk                                      |   3 +
 27 files changed, 2015 insertions(+), 4 deletions(-)
 create mode 100644 doc/guides/prog_guide/metrics_lib.rst
 create mode 100644 lib/librte_bitratestats/Makefile
 create mode 100644 lib/librte_bitratestats/rte_bitrate.c
 create mode 100644 lib/librte_bitratestats/rte_bitrate.h
 create mode 100644 lib/librte_bitratestats/rte_bitratestats_version.map
 create mode 100644 lib/librte_latencystats/Makefile
 create mode 100644 lib/librte_latencystats/rte_latencystats.c
 create mode 100644 lib/librte_latencystats/rte_latencystats.h
 create mode 100644 lib/librte_latencystats/rte_latencystats_version.map
 create mode 100644 lib/librte_metrics/Makefile
 create mode 100644 lib/librte_metrics/rte_metrics.c
 create mode 100644 lib/librte_metrics/rte_metrics.h
 create mode 100644 lib/librte_metrics/rte_metrics_version.map


More information about the dev mailing list