[dpdk-dev] [RFC 0/6] replace telemetry with process_info

[Bruce Richardson]

On Wed, Feb 05, 2020 at 04:21:16PM +0100, David Marchand wrote:
> Hello Ciara, Bruce,
> 
> On Thu, Dec 5, 2019 at 6:34 PM Ciara Power <ciara.power at intel.com> wrote:
> >
> > From: Bruce Richardson <bruce.richardson at intel.com>
> >
> > This patchset proposes a new library, called "process-info" for now, to
> > replace the existing telemetry library in DPDK. (Name subject to change
> > if someone can propose a better one).
> >
> > The existing telemetry library provides useful capabilities if used:
> >   - Creates a unix socket on the system to allow external programs
> >     connect and gather stats about the DPDK process.
> >   - Supports outputting the xstats for various network cards on the
> >     system.
> >   - Can be used to output any other information exported to the metrics
> >     library, e.g. by applications.
> >   - Uses JSON message format, which is widely supported by other
> >     languages and systems.
> >   - Is supported by a plugin to collectd.
> >
> > However, the library also has some issues and limitations that could be
> > improved upon:
> >   - Has a dependency on libjansson for JSON processing, so is disabled
> >     by default.
> >   - Tied entirely to the metrics library for statistics.
> >   - No support for sending non-stats data, e.g. something as simple as
> >     DPDK version string.
> >   - All data gathering functions are in the library itself, which also
> >     means…
> >   - No support for libraries or drivers to present their own
> >     information via the library.
> >
> > We therefore propose to keep the good points of the existing library,
> > but change the way things work to rectify the downsides.
> > This leads to the following design choices in the library:
> >   - Keep the existing idea of using a unix socket for connection (just
> >     simplifying the connection handling).
> >   - We would like to use JSON format, where possible, but the jansson
> >     library dependency is problematic. However, creating JSON-encoded
> >     data is easier than trying to parse JSON in C code, so we can keep
> >     the JSON output format for processing by e.g. collectd and other
> >     tools, while simplifying the input to be plain text commands:
> >         - Commands in this RFC are designed to all start with "/" for
> >           consistency
> >         - Any parameters to the commands, e.g. the specific port to get
> >           stats for, are placed after a comma ","
> >   - Have the library only handle socket creation and input handling.
> >     All data gathering should be provided by functions external to the
> >     library registered by other components, e.g. have ethdev library
> >     provide the function to query NIC xstats, etc.
> >   - Have the library directly initialized by EAL by default, so that
> >     unless an app explicitly does not want the support, monitoring is
> >     available on all DPDK apps.
> >
> > The obvious question that remains to be answered here is: "why a new
> > library rather than just fixing the old one?"
> >
> > The answer is that we have conflicts between the final two design
> > choices above, which require that the library be built early in the
> > build as other libraries will call into it to register callbacks, and
> > the desire to keep backward compatibility e.g. for use with collectd
> > plugin, which requires the existing library code be kept around and
> > built - as it is now - at the end of the build process since it calls
> > into other DPDK libraries. We therefore cannot have one library that
> > meets both requirements, hence the replacement which allows us to
> > maintain backward compatibility by just leaving the old lib in place
> > until e.g. 20.11.
> >
> > This is also why the new library is called "process_info", since the
> > name "telemetry" is already taken. Suggestions for a better name
> > welcome.
> 
> The only user of the rte_telemetry api I could find is the (not yet
> merged [1]) dpdk collectd plugin.
> 
> How will this impact it?
> Can we expect compatibility?
> 
> 
> 1: https://github.com/collectd/collectd/pull/3273
> 
Yes, we are aware of this, and we are investigating compatibility options.
Hopefully, we'll have more on this to share in 20.05 timeframe, as we do
more prototyping and investigation.

/Bruce