[dpdk-dev] [PATCH v4 1/4] doc: separate versioning.rst into version and policy
Aaron Conole
aconole at redhat.com
Fri Sep 27 17:43:36 CEST 2019
Ray Kinsella <mdr at ashroe.eu> writes:
> Separate versioning.rst into abi versioning and abi policy guidance, in
> preparation for adding more detail to the abi policy.
>
> Signed-off-by: Ray Kinsella <mdr at ashroe.eu>
> ---
> doc/guides/contributing/abi_policy.rst | 169 +++++++++
> doc/guides/contributing/abi_versioning.rst | 427 +++++++++++++++++++++
> doc/guides/contributing/index.rst | 3 +-
> doc/guides/contributing/versioning.rst | 591 -----------------------------
> 4 files changed, 598 insertions(+), 592 deletions(-)
> create mode 100644 doc/guides/contributing/abi_policy.rst
> create mode 100644 doc/guides/contributing/abi_versioning.rst
> delete mode 100644 doc/guides/contributing/versioning.rst
>
> diff --git a/doc/guides/contributing/abi_policy.rst b/doc/guides/contributing/abi_policy.rst
> new file mode 100644
> index 0000000..55bacb4
> --- /dev/null
> +++ b/doc/guides/contributing/abi_policy.rst
> @@ -0,0 +1,169 @@
> +.. SPDX-License-Identifier: BSD-3-Clause
> + Copyright 2018 The DPDK contributors
> +
> +.. abi_api_policy:
> +
> +DPDK ABI/API policy
> +===================
> +
> +Description
> +-----------
> +
> +This document details some methods for handling ABI management in the DPDK.
> +
> +General Guidelines
> +------------------
> +
> +#. Whenever possible, ABI should be preserved
> +#. ABI/API may be changed with a deprecation process
> +#. The modification of symbols can generally be managed with versioning
> +#. Libraries or APIs marked in ``experimental`` state may change without constraint
> +#. New APIs will be marked as ``experimental`` for at least one release to allow
> + any issues found by users of the new API to be fixed quickly
> +#. The addition of symbols is generally not problematic
> +#. The removal of symbols generally is an ABI break and requires bumping of the
> + LIBABIVER macro
> +#. Updates to the minimum hardware requirements, which drop support for hardware which
> + was previously supported, should be treated as an ABI change.
> +
> +What is an ABI
> +~~~~~~~~~~~~~~
> +
> +An ABI (Application Binary Interface) is the set of runtime interfaces exposed
> +by a library. It is similar to an API (Application Programming Interface) but
> +is the result of compilation. It is also effectively cloned when applications
> +link to dynamic libraries. That is to say when an application is compiled to
> +link against dynamic libraries, it is assumed that the ABI remains constant
> +between the time the application is compiled/linked, and the time that it runs.
> +Therefore, in the case of dynamic linking, it is critical that an ABI is
> +preserved, or (when modified), done in such a way that the application is unable
> +to behave improperly or in an unexpected fashion.
> +
This section probably needs a bit more details. People still are
confused what exactly constitutes ABI vs. API (see
http://mails.dpdk.org/archives/dev/2018-January/085209.html for a
confusing example).
It's important that people know not just function signatures, but also
return codes, and even data structure layouts are all part of the ABI.
> +
> +ABI/API Deprecation
> +-------------------
> +
> +The DPDK ABI policy
> +~~~~~~~~~~~~~~~~~~~
> +
> +ABI versions are set at the time of major release labeling, and the ABI may
> +change multiple times, without warning, between the last release label and the
> +HEAD label of the git tree.
> +
> +ABI versions, once released, are available until such time as their
> +deprecation has been noted in the Release Notes for at least one major release
> +cycle. For example consider the case where the ABI for DPDK 2.0 has been
> +shipped and then a decision is made to modify it during the development of
> +DPDK 2.1. The decision will be recorded in the Release Notes for the DPDK 2.1
> +release and the modification will be made available in the DPDK 2.2 release.
> +
> +ABI versions may be deprecated in whole or in part as needed by a given
> +update.
> +
> +Some ABI changes may be too significant to reasonably maintain multiple
> +versions. In those cases ABI's may be updated without backward compatibility
> +being provided. The requirements for doing so are:
> +
> +#. At least 3 acknowledgments of the need to do so must be made on the
> + dpdk.org mailing list.
> +
> + - The acknowledgment of the maintainer of the component is mandatory, or if
> + no maintainer is available for the component, the tree/sub-tree maintainer
> + for that component must acknowledge the ABI change instead.
> +
> + - It is also recommended that acknowledgments from different "areas of
> + interest" be sought for each deprecation, for example: from NIC vendors,
> + CPU vendors, end-users, etc.
> +
> +#. The changes (including an alternative map file) can be included with
> + deprecation notice, in wrapped way by the ``RTE_NEXT_ABI`` option,
> + to provide more details about oncoming changes.
> + ``RTE_NEXT_ABI`` wrapper will be removed when it become the default ABI.
> + More preferred way to provide this information is sending the feature
> + as a separate patch and reference it in deprecation notice.
> +
> +#. A full deprecation cycle, as explained above, must be made to offer
> + downstream consumers sufficient warning of the change.
> +
> +Note that the above process for ABI deprecation should not be undertaken
> +lightly. ABI stability is extremely important for downstream consumers of the
> +DPDK, especially when distributed in shared object form. Every effort should
> +be made to preserve the ABI whenever possible. The ABI should only be changed
> +for significant reasons, such as performance enhancements. ABI breakage due to
> +changes such as reorganizing public structure fields for aesthetic or
> +readability purposes should be avoided.
> +
> +.. note::
> +
> + Updates to the minimum hardware requirements, which drop support for hardware
> + which was previously supported, should be treated as an ABI change, and
> + follow the relevant deprecation policy procedures as above: 3 acks and
> + announcement at least one release in advance.
> +
> +Examples of Deprecation Notices
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The following are some examples of ABI deprecation notices which would be
> +added to the Release Notes:
> +
> +* The Macro ``#RTE_FOO`` is deprecated and will be removed with version 2.0,
> + to be replaced with the inline function ``rte_foo()``.
> +
> +* The function ``rte_mbuf_grok()`` has been updated to include a new parameter
> + in version 2.0. Backwards compatibility will be maintained for this function
> + until the release of version 2.1
> +
> +* The members of ``struct rte_foo`` have been reorganized in release 2.0 for
> + performance reasons. Existing binary applications will have backwards
> + compatibility in release 2.0, while newly built binaries will need to
> + reference the new structure variant ``struct rte_foo2``. Compatibility will
> + be removed in release 2.2, and all applications will require updating and
> + rebuilding to the new structure at that time, which will be renamed to the
> + original ``struct rte_foo``.
> +
> +* Significant ABI changes are planned for the ``librte_dostuff`` library. The
> + upcoming release 2.0 will not contain these changes, but release 2.1 will,
> + and no backwards compatibility is planned due to the extensive nature of
> + these changes. Binaries using this library built prior to version 2.1 will
> + require updating and recompilation.
> +
> +New API replacing previous one
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +If a new API proposed functionally replaces an existing one, when the new API
> +becomes non-experimental then the old one is marked with ``__rte_deprecated``.
> +Deprecated APIs are removed completely just after the next LTS.
> +
> +Reminder that old API should follow deprecation process to be removed.
> +
> +
> +Experimental APIs
> +-----------------
> +
> +APIs marked as ``experimental`` are not considered part of the ABI and may
> +change without warning at any time. Since changes to APIs are most likely
> +immediately after their introduction, as users begin to take advantage of
> +those new APIs and start finding issues with them, new DPDK APIs will be
> +automatically marked as ``experimental`` to allow for a period of stabilization
> +before they become part of a tracked ABI.
> +
> +Note that marking an API as experimental is a multi step process.
> +To mark an API as experimental, the symbols which are desired to be exported
> +must be placed in an EXPERIMENTAL version block in the corresponding libraries'
> +version map script.
> +Secondly, the corresponding prototypes of those exported functions (in the
> +development header files), must be marked with the ``__rte_experimental`` tag
> +(see ``rte_compat.h``).
> +The DPDK build makefiles perform a check to ensure that the map file and the
> +C code reflect the same list of symbols.
> +This check can be circumvented by defining ``ALLOW_EXPERIMENTAL_API``
> +during compilation in the corresponding library Makefile.
> +
> +In addition to tagging the code with ``__rte_experimental``,
> +the doxygen markup must also contain the EXPERIMENTAL string,
> +and the MAINTAINERS file should note the EXPERIMENTAL libraries.
> +
> +For removing the experimental tag associated with an API, deprecation notice
> +is not required. Though, an API should remain in experimental state for at least
> +one release. Thereafter, normal process of posting patch for review to mailing
> +list can be followed.
> diff --git a/doc/guides/contributing/abi_versioning.rst b/doc/guides/contributing/abi_versioning.rst
> new file mode 100644
> index 0000000..53e6ac0
> --- /dev/null
> +++ b/doc/guides/contributing/abi_versioning.rst
> @@ -0,0 +1,427 @@
> +.. SPDX-License-Identifier: BSD-3-Clause
> + Copyright 2018 The DPDK contributors
> +
> +.. library_versioning:
> +
> +Library versioning
> +------------------
> +
> +Downstreams might want to provide different DPDK releases at the same time to
> +support multiple consumers of DPDK linked against older and newer sonames.
> +
> +Also due to the interdependencies that DPDK libraries can have applications
> +might end up with an executable space in which multiple versions of a library
> +are mapped by ld.so.
> +
> +Think of LibA that got an ABI bump and LibB that did not get an ABI bump but is
> +depending on LibA.
> +
> +.. note::
> +
> + Application
> + \-> LibA.old
> + \-> LibB.new -> LibA.new
> +
> +That is a conflict which can be avoided by setting ``CONFIG_RTE_MAJOR_ABI``.
> +If set, the value of ``CONFIG_RTE_MAJOR_ABI`` overwrites all - otherwise per
> +library - versions defined in the libraries ``LIBABIVER``.
> +An example might be ``CONFIG_RTE_MAJOR_ABI=16.11`` which will make all libraries
> +``librte<?>.so.16.11`` instead of ``librte<?>.so.<LIBABIVER>``.
> +
> +
> +ABI versioning
> +--------------
> +
> +Versioning Macros
> +~~~~~~~~~~~~~~~~~
> +
> +When a symbol is exported from a library to provide an API, it also provides a
> +calling convention (ABI) that is embodied in its name, return type and
> +arguments. Occasionally that function may need to change to accommodate new
> +functionality or behavior. When that occurs, it is desirable to allow for
> +backward compatibility for a time with older binaries that are dynamically
> +linked to the DPDK.
> +
> +To support backward compatibility the ``rte_compat.h``
> +header file provides macros to use when updating exported functions. These
> +macros are used in conjunction with the ``rte_<library>_version.map`` file for
> +a given library to allow multiple versions of a symbol to exist in a shared
> +library so that older binaries need not be immediately recompiled.
> +
> +The macros exported are:
> +
> +* ``VERSION_SYMBOL(b, e, n)``: Creates a symbol version table entry binding
> + versioned symbol ``b at DPDK_n`` to the internal function ``b_e``.
> +
> +* ``BIND_DEFAULT_SYMBOL(b, e, n)``: Creates a symbol version entry instructing
> + the linker to bind references to symbol ``b`` to the internal symbol
> + ``b_e``.
> +
> +* ``MAP_STATIC_SYMBOL(f, p)``: Declare the prototype ``f``, and map it to the
> + fully qualified function ``p``, so that if a symbol becomes versioned, it
> + can still be mapped back to the public symbol name.
> +
> +Examples of ABI Macro use
> +^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Updating a public API
> +_____________________
> +
> +Assume we have a function as follows
> +
> +.. code-block:: c
> +
> + /*
> + * Create an acl context object for apps to
> + * manipulate
> + */
> + struct rte_acl_ctx *
> + rte_acl_create(const struct rte_acl_param *param)
> + {
> + ...
> + }
> +
> +
> +Assume that struct rte_acl_ctx is a private structure, and that a developer
> +wishes to enhance the acl api so that a debugging flag can be enabled on a
> +per-context basis. This requires an addition to the structure (which, being
> +private, is safe), but it also requires modifying the code as follows
> +
> +.. code-block:: c
> +
> + /*
> + * Create an acl context object for apps to
> + * manipulate
> + */
> + struct rte_acl_ctx *
> + rte_acl_create(const struct rte_acl_param *param, int debug)
> + {
> + ...
> + }
> +
> +
> +Note also that, being a public function, the header file prototype must also be
> +changed, as must all the call sites, to reflect the new ABI footprint. We will
> +maintain previous ABI versions that are accessible only to previously compiled
> +binaries
> +
> +The addition of a parameter to the function is ABI breaking as the function is
> +public, and existing application may use it in its current form. However, the
> +compatibility macros in DPDK allow a developer to use symbol versioning so that
> +multiple functions can be mapped to the same public symbol based on when an
> +application was linked to it. To see how this is done, we start with the
> +requisite libraries version map file. Initially the version map file for the
> +acl library looks like this
> +
> +.. code-block:: none
> +
> + DPDK_2.0 {
> + global:
> +
> + rte_acl_add_rules;
> + rte_acl_build;
> + rte_acl_classify;
> + rte_acl_classify_alg;
> + rte_acl_classify_scalar;
> + rte_acl_create;
> + rte_acl_dump;
> + rte_acl_find_existing;
> + rte_acl_free;
> + rte_acl_ipv4vlan_add_rules;
> + rte_acl_ipv4vlan_build;
> + rte_acl_list_dump;
> + rte_acl_reset;
> + rte_acl_reset_rules;
> + rte_acl_set_ctx_classify;
> +
> + local: *;
> + };
> +
> +This file needs to be modified as follows
> +
> +.. code-block:: none
> +
> + DPDK_2.0 {
> + global:
> +
> + rte_acl_add_rules;
> + rte_acl_build;
> + rte_acl_classify;
> + rte_acl_classify_alg;
> + rte_acl_classify_scalar;
> + rte_acl_create;
> + rte_acl_dump;
> + rte_acl_find_existing;
> + rte_acl_free;
> + rte_acl_ipv4vlan_add_rules;
> + rte_acl_ipv4vlan_build;
> + rte_acl_list_dump;
> + rte_acl_reset;
> + rte_acl_reset_rules;
> + rte_acl_set_ctx_classify;
> +
> + local: *;
> + };
> +
> + DPDK_2.1 {
> + global:
> + rte_acl_create;
> +
> + } DPDK_2.0;
> +
> +The addition of the new block tells the linker that a new version node is
> +available (DPDK_2.1), which contains the symbol rte_acl_create, and inherits the
> +symbols from the DPDK_2.0 node. This list is directly translated into a list of
> +exported symbols when DPDK is compiled as a shared library
> +
> +Next, we need to specify in the code which function map to the rte_acl_create
> +symbol at which versions. First, at the site of the initial symbol definition,
> +we need to update the function so that it is uniquely named, and not in conflict
> +with the public symbol name
> +
> +.. code-block:: c
> +
> + struct rte_acl_ctx *
> + -rte_acl_create(const struct rte_acl_param *param)
> + +rte_acl_create_v20(const struct rte_acl_param *param)
> + {
> + size_t sz;
> + struct rte_acl_ctx *ctx;
> + ...
> +
> +Note that the base name of the symbol was kept intact, as this is conducive to
> +the macros used for versioning symbols. That is our next step, mapping this new
> +symbol name to the initial symbol name at version node 2.0. Immediately after
> +the function, we add this line of code
> +
> +.. code-block:: c
> +
> + VERSION_SYMBOL(rte_acl_create, _v20, 2.0);
> +
> +Remembering to also add the rte_compat.h header to the requisite c file where
> +these changes are being made. The above macro instructs the linker to create a
> +new symbol ``rte_acl_create at DPDK_2.0``, which matches the symbol created in older
> +builds, but now points to the above newly named function. We have now mapped
> +the original rte_acl_create symbol to the original function (but with a new
> +name)
> +
> +Next, we need to create the 2.1 version of the symbol. We create a new function
> +name, with a different suffix, and implement it appropriately
> +
> +.. code-block:: c
> +
> + struct rte_acl_ctx *
> + rte_acl_create_v21(const struct rte_acl_param *param, int debug);
> + {
> + struct rte_acl_ctx *ctx = rte_acl_create_v20(param);
> +
> + ctx->debug = debug;
> +
> + return ctx;
> + }
> +
> +This code serves as our new API call. Its the same as our old call, but adds
> +the new parameter in place. Next we need to map this function to the symbol
> +``rte_acl_create at DPDK_2.1``. To do this, we modify the public prototype of the call
> +in the header file, adding the macro there to inform all including applications,
> +that on re-link, the default rte_acl_create symbol should point to this
> +function. Note that we could do this by simply naming the function above
> +rte_acl_create, and the linker would chose the most recent version tag to apply
> +in the version script, but we can also do this in the header file
> +
> +.. code-block:: c
> +
> + struct rte_acl_ctx *
> + -rte_acl_create(const struct rte_acl_param *param);
> + +rte_acl_create(const struct rte_acl_param *param, int debug);
> + +BIND_DEFAULT_SYMBOL(rte_acl_create, _v21, 2.1);
> +
> +The BIND_DEFAULT_SYMBOL macro explicitly tells applications that include this
> +header, to link to the rte_acl_create_v21 function and apply the DPDK_2.1
> +version node to it. This method is more explicit and flexible than just
> +re-implementing the exact symbol name, and allows for other features (such as
> +linking to the old symbol version by default, when the new ABI is to be opt-in
> +for a period.
> +
> +One last thing we need to do. Note that we've taken what was a public symbol,
> +and duplicated it into two uniquely and differently named symbols. We've then
> +mapped each of those back to the public symbol ``rte_acl_create`` with different
> +version tags. This only applies to dynamic linking, as static linking has no
> +notion of versioning. That leaves this code in a position of no longer having a
> +symbol simply named ``rte_acl_create`` and a static build will fail on that
> +missing symbol.
> +
> +To correct this, we can simply map a function of our choosing back to the public
> +symbol in the static build with the ``MAP_STATIC_SYMBOL`` macro. Generally the
> +assumption is that the most recent version of the symbol is the one you want to
> +map. So, back in the C file where, immediately after ``rte_acl_create_v21`` is
> +defined, we add this
> +
> +.. code-block:: c
> +
> + struct rte_acl_ctx *
> + rte_acl_create_v21(const struct rte_acl_param *param, int debug)
> + {
> + ...
> + }
> + MAP_STATIC_SYMBOL(struct rte_acl_ctx *rte_acl_create(const struct rte_acl_param *param, int debug), rte_acl_create_v21);
> +
> +That tells the compiler that, when building a static library, any calls to the
> +symbol ``rte_acl_create`` should be linked to ``rte_acl_create_v21``
> +
> +That's it, on the next shared library rebuild, there will be two versions of
> +rte_acl_create, an old DPDK_2.0 version, used by previously built applications,
> +and a new DPDK_2.1 version, used by future built applications.
> +
> +
> +Deprecating part of a public API
> +________________________________
> +
> +Lets assume that you've done the above update, and after a few releases have
> +passed you decide you would like to retire the old version of the function.
> +After having gone through the ABI deprecation announcement process, removal is
> +easy. Start by removing the symbol from the requisite version map file:
> +
> +.. code-block:: none
> +
> + DPDK_2.0 {
> + global:
> +
> + rte_acl_add_rules;
> + rte_acl_build;
> + rte_acl_classify;
> + rte_acl_classify_alg;
> + rte_acl_classify_scalar;
> + rte_acl_dump;
> + - rte_acl_create
> + rte_acl_find_existing;
> + rte_acl_free;
> + rte_acl_ipv4vlan_add_rules;
> + rte_acl_ipv4vlan_build;
> + rte_acl_list_dump;
> + rte_acl_reset;
> + rte_acl_reset_rules;
> + rte_acl_set_ctx_classify;
> +
> + local: *;
> + };
> +
> + DPDK_2.1 {
> + global:
> + rte_acl_create;
> + } DPDK_2.0;
> +
> +
> +Next remove the corresponding versioned export.
> +
> +.. code-block:: c
> +
> + -VERSION_SYMBOL(rte_acl_create, _v20, 2.0);
> +
> +
> +Note that the internal function definition could also be removed, but its used
> +in our example by the newer version _v21, so we leave it in place. This is a
> +coding style choice.
> +
> +Lastly, we need to bump the LIBABIVER number for this library in the Makefile to
> +indicate to applications doing dynamic linking that this is a later, and
> +possibly incompatible library version:
> +
> +.. code-block:: c
> +
> + -LIBABIVER := 1
> + +LIBABIVER := 2
> +
> +Deprecating an entire ABI version
> +_________________________________
> +
> +While removing a symbol from and ABI may be useful, it is often more practical
> +to remove an entire version node at once. If a version node completely
> +specifies an API, then removing part of it, typically makes it incomplete. In
> +those cases it is better to remove the entire node
> +
> +To do this, start by modifying the version map file, such that all symbols from
> +the node to be removed are merged into the next node in the map
> +
> +In the case of our map above, it would transform to look as follows
> +
> +.. code-block:: none
> +
> + DPDK_2.1 {
> + global:
> +
> + rte_acl_add_rules;
> + rte_acl_build;
> + rte_acl_classify;
> + rte_acl_classify_alg;
> + rte_acl_classify_scalar;
> + rte_acl_dump;
> + rte_acl_create
> + rte_acl_find_existing;
> + rte_acl_free;
> + rte_acl_ipv4vlan_add_rules;
> + rte_acl_ipv4vlan_build;
> + rte_acl_list_dump;
> + rte_acl_reset;
> + rte_acl_reset_rules;
> + rte_acl_set_ctx_classify;
> +
> + local: *;
> + };
> +
> +Then any uses of BIND_DEFAULT_SYMBOL that pointed to the old node should be
> +updated to point to the new version node in any header files for all affected
> +symbols.
> +
> +.. code-block:: c
> +
> + -BIND_DEFAULT_SYMBOL(rte_acl_create, _v20, 2.0);
> + +BIND_DEFAULT_SYMBOL(rte_acl_create, _v21, 2.1);
> +
> +Lastly, any VERSION_SYMBOL macros that point to the old version node should be
> +removed, taking care to keep, where need old code in place to support newer
> +versions of the symbol.
> +
> +
> +Running the ABI Validator
> +-------------------------
> +
> +The ``devtools`` directory in the DPDK source tree contains a utility program,
> +``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> +Compliance Checker
> +<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> +
> +This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> +utilities which can be installed via a package manager. For example::
> +
> + sudo yum install abi-compliance-checker
> + sudo yum install abi-dumper
> +
> +The syntax of the ``validate-abi.sh`` utility is::
> +
> + ./devtools/validate-abi.sh <REV1> <REV2>
> +
> +Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
> +https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
> +on the local repo.
> +
> +For example::
> +
> + # Check between the previous and latest commit:
> + ./devtools/validate-abi.sh HEAD~1 HEAD
> +
> + # Check on a specific compilation target:
> + ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
> +
> + # Check between two tags:
> + ./devtools/validate-abi.sh v2.0.0 v2.1.0
> +
> + # Check between git master and local topic-branch "vhost-hacking":
> + ./devtools/validate-abi.sh master vhost-hacking
> +
> +After the validation script completes (it can take a while since it need to
> +compile both tags) it will create compatibility reports in the
> +``./abi-check/compat_report`` directory. Listed incompatibilities can be found
> +as follows::
> +
> + grep -lr Incompatible abi-check/compat_reports/
> diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
> index e2608d3..2fefd91 100644
> --- a/doc/guides/contributing/index.rst
> +++ b/doc/guides/contributing/index.rst
> @@ -10,7 +10,8 @@ Contributor's Guidelines
>
> coding_style
> design
> - versioning
> + abi_policy
> + abi_versioning
> documentation
> patches
> vulnerability
> diff --git a/doc/guides/contributing/versioning.rst b/doc/guides/contributing/versioning.rst
> deleted file mode 100644
> index 3ab2c43..0000000
> --- a/doc/guides/contributing/versioning.rst
> +++ /dev/null
> @@ -1,591 +0,0 @@
> -.. SPDX-License-Identifier: BSD-3-Clause
> - Copyright 2018 The DPDK contributors
> -
> -DPDK ABI/API policy
> -===================
> -
> -Description
> ------------
> -
> -This document details some methods for handling ABI management in the DPDK.
> -
> -General Guidelines
> -------------------
> -
> -#. Whenever possible, ABI should be preserved
> -#. ABI/API may be changed with a deprecation process
> -#. The modification of symbols can generally be managed with versioning
> -#. Libraries or APIs marked in ``experimental`` state may change without constraint
> -#. New APIs will be marked as ``experimental`` for at least one release to allow
> - any issues found by users of the new API to be fixed quickly
> -#. The addition of symbols is generally not problematic
> -#. The removal of symbols generally is an ABI break and requires bumping of the
> - LIBABIVER macro
> -#. Updates to the minimum hardware requirements, which drop support for hardware which
> - was previously supported, should be treated as an ABI change.
> -
> -What is an ABI
> -~~~~~~~~~~~~~~
> -
> -An ABI (Application Binary Interface) is the set of runtime interfaces exposed
> -by a library. It is similar to an API (Application Programming Interface) but
> -is the result of compilation. It is also effectively cloned when applications
> -link to dynamic libraries. That is to say when an application is compiled to
> -link against dynamic libraries, it is assumed that the ABI remains constant
> -between the time the application is compiled/linked, and the time that it runs.
> -Therefore, in the case of dynamic linking, it is critical that an ABI is
> -preserved, or (when modified), done in such a way that the application is unable
> -to behave improperly or in an unexpected fashion.
> -
> -
> -ABI/API Deprecation
> --------------------
> -
> -The DPDK ABI policy
> -~~~~~~~~~~~~~~~~~~~
> -
> -ABI versions are set at the time of major release labeling, and the ABI may
> -change multiple times, without warning, between the last release label and the
> -HEAD label of the git tree.
> -
> -ABI versions, once released, are available until such time as their
> -deprecation has been noted in the Release Notes for at least one major release
> -cycle. For example consider the case where the ABI for DPDK 2.0 has been
> -shipped and then a decision is made to modify it during the development of
> -DPDK 2.1. The decision will be recorded in the Release Notes for the DPDK 2.1
> -release and the modification will be made available in the DPDK 2.2 release.
> -
> -ABI versions may be deprecated in whole or in part as needed by a given
> -update.
> -
> -Some ABI changes may be too significant to reasonably maintain multiple
> -versions. In those cases ABI's may be updated without backward compatibility
> -being provided. The requirements for doing so are:
> -
> -#. At least 3 acknowledgments of the need to do so must be made on the
> - dpdk.org mailing list.
> -
> - - The acknowledgment of the maintainer of the component is mandatory, or if
> - no maintainer is available for the component, the tree/sub-tree maintainer
> - for that component must acknowledge the ABI change instead.
> -
> - - It is also recommended that acknowledgments from different "areas of
> - interest" be sought for each deprecation, for example: from NIC vendors,
> - CPU vendors, end-users, etc.
> -
> -#. The changes (including an alternative map file) can be included with
> - deprecation notice, in wrapped way by the ``RTE_NEXT_ABI`` option,
> - to provide more details about oncoming changes.
> - ``RTE_NEXT_ABI`` wrapper will be removed when it become the default ABI.
> - More preferred way to provide this information is sending the feature
> - as a separate patch and reference it in deprecation notice.
> -
> -#. A full deprecation cycle, as explained above, must be made to offer
> - downstream consumers sufficient warning of the change.
> -
> -Note that the above process for ABI deprecation should not be undertaken
> -lightly. ABI stability is extremely important for downstream consumers of the
> -DPDK, especially when distributed in shared object form. Every effort should
> -be made to preserve the ABI whenever possible. The ABI should only be changed
> -for significant reasons, such as performance enhancements. ABI breakage due to
> -changes such as reorganizing public structure fields for aesthetic or
> -readability purposes should be avoided.
> -
> -.. note::
> -
> - Updates to the minimum hardware requirements, which drop support for hardware
> - which was previously supported, should be treated as an ABI change, and
> - follow the relevant deprecation policy procedures as above: 3 acks and
> - announcement at least one release in advance.
> -
> -Examples of Deprecation Notices
> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> -
> -The following are some examples of ABI deprecation notices which would be
> -added to the Release Notes:
> -
> -* The Macro ``#RTE_FOO`` is deprecated and will be removed with version 2.0,
> - to be replaced with the inline function ``rte_foo()``.
> -
> -* The function ``rte_mbuf_grok()`` has been updated to include a new parameter
> - in version 2.0. Backwards compatibility will be maintained for this function
> - until the release of version 2.1
> -
> -* The members of ``struct rte_foo`` have been reorganized in release 2.0 for
> - performance reasons. Existing binary applications will have backwards
> - compatibility in release 2.0, while newly built binaries will need to
> - reference the new structure variant ``struct rte_foo2``. Compatibility will
> - be removed in release 2.2, and all applications will require updating and
> - rebuilding to the new structure at that time, which will be renamed to the
> - original ``struct rte_foo``.
> -
> -* Significant ABI changes are planned for the ``librte_dostuff`` library. The
> - upcoming release 2.0 will not contain these changes, but release 2.1 will,
> - and no backwards compatibility is planned due to the extensive nature of
> - these changes. Binaries using this library built prior to version 2.1 will
> - require updating and recompilation.
> -
> -New API replacing previous one
> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> -
> -If a new API proposed functionally replaces an existing one, when the new API
> -becomes non-experimental then the old one is marked with ``__rte_deprecated``.
> -Deprecated APIs are removed completely just after the next LTS.
> -
> -Reminder that old API should follow deprecation process to be removed.
> -
> -
> -Experimental APIs
> ------------------
> -
> -APIs marked as ``experimental`` are not considered part of the ABI and may
> -change without warning at any time. Since changes to APIs are most likely
> -immediately after their introduction, as users begin to take advantage of
> -those new APIs and start finding issues with them, new DPDK APIs will be
> -automatically marked as ``experimental`` to allow for a period of stabilization
> -before they become part of a tracked ABI.
> -
> -Note that marking an API as experimental is a multi step process.
> -To mark an API as experimental, the symbols which are desired to be exported
> -must be placed in an EXPERIMENTAL version block in the corresponding libraries'
> -version map script.
> -Secondly, the corresponding prototypes of those exported functions (in the
> -development header files), must be marked with the ``__rte_experimental`` tag
> -(see ``rte_compat.h``).
> -The DPDK build makefiles perform a check to ensure that the map file and the
> -C code reflect the same list of symbols.
> -This check can be circumvented by defining ``ALLOW_EXPERIMENTAL_API``
> -during compilation in the corresponding library Makefile.
> -
> -In addition to tagging the code with ``__rte_experimental``,
> -the doxygen markup must also contain the EXPERIMENTAL string,
> -and the MAINTAINERS file should note the EXPERIMENTAL libraries.
> -
> -For removing the experimental tag associated with an API, deprecation notice
> -is not required. Though, an API should remain in experimental state for at least
> -one release. Thereafter, normal process of posting patch for review to mailing
> -list can be followed.
> -
> -
> -Library versioning
> -------------------
> -
> -Downstreams might want to provide different DPDK releases at the same time to
> -support multiple consumers of DPDK linked against older and newer sonames.
> -
> -Also due to the interdependencies that DPDK libraries can have applications
> -might end up with an executable space in which multiple versions of a library
> -are mapped by ld.so.
> -
> -Think of LibA that got an ABI bump and LibB that did not get an ABI bump but is
> -depending on LibA.
> -
> -.. note::
> -
> - Application
> - \-> LibA.old
> - \-> LibB.new -> LibA.new
> -
> -That is a conflict which can be avoided by setting ``CONFIG_RTE_MAJOR_ABI``.
> -If set, the value of ``CONFIG_RTE_MAJOR_ABI`` overwrites all - otherwise per
> -library - versions defined in the libraries ``LIBABIVER``.
> -An example might be ``CONFIG_RTE_MAJOR_ABI=16.11`` which will make all libraries
> -``librte<?>.so.16.11`` instead of ``librte<?>.so.<LIBABIVER>``.
> -
> -
> -ABI versioning
> ---------------
> -
> -Versioning Macros
> -~~~~~~~~~~~~~~~~~
> -
> -When a symbol is exported from a library to provide an API, it also provides a
> -calling convention (ABI) that is embodied in its name, return type and
> -arguments. Occasionally that function may need to change to accommodate new
> -functionality or behavior. When that occurs, it is desirable to allow for
> -backward compatibility for a time with older binaries that are dynamically
> -linked to the DPDK.
> -
> -To support backward compatibility the ``rte_compat.h``
> -header file provides macros to use when updating exported functions. These
> -macros are used in conjunction with the ``rte_<library>_version.map`` file for
> -a given library to allow multiple versions of a symbol to exist in a shared
> -library so that older binaries need not be immediately recompiled.
> -
> -The macros exported are:
> -
> -* ``VERSION_SYMBOL(b, e, n)``: Creates a symbol version table entry binding
> - versioned symbol ``b at DPDK_n`` to the internal function ``b_e``.
> -
> -* ``BIND_DEFAULT_SYMBOL(b, e, n)``: Creates a symbol version entry instructing
> - the linker to bind references to symbol ``b`` to the internal symbol
> - ``b_e``.
> -
> -* ``MAP_STATIC_SYMBOL(f, p)``: Declare the prototype ``f``, and map it to the
> - fully qualified function ``p``, so that if a symbol becomes versioned, it
> - can still be mapped back to the public symbol name.
> -
> -Examples of ABI Macro use
> -^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -Updating a public API
> -_____________________
> -
> -Assume we have a function as follows
> -
> -.. code-block:: c
> -
> - /*
> - * Create an acl context object for apps to
> - * manipulate
> - */
> - struct rte_acl_ctx *
> - rte_acl_create(const struct rte_acl_param *param)
> - {
> - ...
> - }
> -
> -
> -Assume that struct rte_acl_ctx is a private structure, and that a developer
> -wishes to enhance the acl api so that a debugging flag can be enabled on a
> -per-context basis. This requires an addition to the structure (which, being
> -private, is safe), but it also requires modifying the code as follows
> -
> -.. code-block:: c
> -
> - /*
> - * Create an acl context object for apps to
> - * manipulate
> - */
> - struct rte_acl_ctx *
> - rte_acl_create(const struct rte_acl_param *param, int debug)
> - {
> - ...
> - }
> -
> -
> -Note also that, being a public function, the header file prototype must also be
> -changed, as must all the call sites, to reflect the new ABI footprint. We will
> -maintain previous ABI versions that are accessible only to previously compiled
> -binaries
> -
> -The addition of a parameter to the function is ABI breaking as the function is
> -public, and existing application may use it in its current form. However, the
> -compatibility macros in DPDK allow a developer to use symbol versioning so that
> -multiple functions can be mapped to the same public symbol based on when an
> -application was linked to it. To see how this is done, we start with the
> -requisite libraries version map file. Initially the version map file for the
> -acl library looks like this
> -
> -.. code-block:: none
> -
> - DPDK_2.0 {
> - global:
> -
> - rte_acl_add_rules;
> - rte_acl_build;
> - rte_acl_classify;
> - rte_acl_classify_alg;
> - rte_acl_classify_scalar;
> - rte_acl_create;
> - rte_acl_dump;
> - rte_acl_find_existing;
> - rte_acl_free;
> - rte_acl_ipv4vlan_add_rules;
> - rte_acl_ipv4vlan_build;
> - rte_acl_list_dump;
> - rte_acl_reset;
> - rte_acl_reset_rules;
> - rte_acl_set_ctx_classify;
> -
> - local: *;
> - };
> -
> -This file needs to be modified as follows
> -
> -.. code-block:: none
> -
> - DPDK_2.0 {
> - global:
> -
> - rte_acl_add_rules;
> - rte_acl_build;
> - rte_acl_classify;
> - rte_acl_classify_alg;
> - rte_acl_classify_scalar;
> - rte_acl_create;
> - rte_acl_dump;
> - rte_acl_find_existing;
> - rte_acl_free;
> - rte_acl_ipv4vlan_add_rules;
> - rte_acl_ipv4vlan_build;
> - rte_acl_list_dump;
> - rte_acl_reset;
> - rte_acl_reset_rules;
> - rte_acl_set_ctx_classify;
> -
> - local: *;
> - };
> -
> - DPDK_2.1 {
> - global:
> - rte_acl_create;
> -
> - } DPDK_2.0;
> -
> -The addition of the new block tells the linker that a new version node is
> -available (DPDK_2.1), which contains the symbol rte_acl_create, and inherits the
> -symbols from the DPDK_2.0 node. This list is directly translated into a list of
> -exported symbols when DPDK is compiled as a shared library
> -
> -Next, we need to specify in the code which function map to the rte_acl_create
> -symbol at which versions. First, at the site of the initial symbol definition,
> -we need to update the function so that it is uniquely named, and not in conflict
> -with the public symbol name
> -
> -.. code-block:: c
> -
> - struct rte_acl_ctx *
> - -rte_acl_create(const struct rte_acl_param *param)
> - +rte_acl_create_v20(const struct rte_acl_param *param)
> - {
> - size_t sz;
> - struct rte_acl_ctx *ctx;
> - ...
> -
> -Note that the base name of the symbol was kept intact, as this is conducive to
> -the macros used for versioning symbols. That is our next step, mapping this new
> -symbol name to the initial symbol name at version node 2.0. Immediately after
> -the function, we add this line of code
> -
> -.. code-block:: c
> -
> - VERSION_SYMBOL(rte_acl_create, _v20, 2.0);
> -
> -Remembering to also add the rte_compat.h header to the requisite c file where
> -these changes are being made. The above macro instructs the linker to create a
> -new symbol ``rte_acl_create at DPDK_2.0``, which matches the symbol created in older
> -builds, but now points to the above newly named function. We have now mapped
> -the original rte_acl_create symbol to the original function (but with a new
> -name)
> -
> -Next, we need to create the 2.1 version of the symbol. We create a new function
> -name, with a different suffix, and implement it appropriately
> -
> -.. code-block:: c
> -
> - struct rte_acl_ctx *
> - rte_acl_create_v21(const struct rte_acl_param *param, int debug);
> - {
> - struct rte_acl_ctx *ctx = rte_acl_create_v20(param);
> -
> - ctx->debug = debug;
> -
> - return ctx;
> - }
> -
> -This code serves as our new API call. Its the same as our old call, but adds
> -the new parameter in place. Next we need to map this function to the symbol
> -``rte_acl_create at DPDK_2.1``. To do this, we modify the public prototype of the call
> -in the header file, adding the macro there to inform all including applications,
> -that on re-link, the default rte_acl_create symbol should point to this
> -function. Note that we could do this by simply naming the function above
> -rte_acl_create, and the linker would chose the most recent version tag to apply
> -in the version script, but we can also do this in the header file
> -
> -.. code-block:: c
> -
> - struct rte_acl_ctx *
> - -rte_acl_create(const struct rte_acl_param *param);
> - +rte_acl_create(const struct rte_acl_param *param, int debug);
> - +BIND_DEFAULT_SYMBOL(rte_acl_create, _v21, 2.1);
> -
> -The BIND_DEFAULT_SYMBOL macro explicitly tells applications that include this
> -header, to link to the rte_acl_create_v21 function and apply the DPDK_2.1
> -version node to it. This method is more explicit and flexible than just
> -re-implementing the exact symbol name, and allows for other features (such as
> -linking to the old symbol version by default, when the new ABI is to be opt-in
> -for a period.
> -
> -One last thing we need to do. Note that we've taken what was a public symbol,
> -and duplicated it into two uniquely and differently named symbols. We've then
> -mapped each of those back to the public symbol ``rte_acl_create`` with different
> -version tags. This only applies to dynamic linking, as static linking has no
> -notion of versioning. That leaves this code in a position of no longer having a
> -symbol simply named ``rte_acl_create`` and a static build will fail on that
> -missing symbol.
> -
> -To correct this, we can simply map a function of our choosing back to the public
> -symbol in the static build with the ``MAP_STATIC_SYMBOL`` macro. Generally the
> -assumption is that the most recent version of the symbol is the one you want to
> -map. So, back in the C file where, immediately after ``rte_acl_create_v21`` is
> -defined, we add this
> -
> -.. code-block:: c
> -
> - struct rte_acl_ctx *
> - rte_acl_create_v21(const struct rte_acl_param *param, int debug)
> - {
> - ...
> - }
> - MAP_STATIC_SYMBOL(struct rte_acl_ctx *rte_acl_create(const struct rte_acl_param *param, int debug), rte_acl_create_v21);
> -
> -That tells the compiler that, when building a static library, any calls to the
> -symbol ``rte_acl_create`` should be linked to ``rte_acl_create_v21``
> -
> -That's it, on the next shared library rebuild, there will be two versions of
> -rte_acl_create, an old DPDK_2.0 version, used by previously built applications,
> -and a new DPDK_2.1 version, used by future built applications.
> -
> -
> -Deprecating part of a public API
> -________________________________
> -
> -Lets assume that you've done the above update, and after a few releases have
> -passed you decide you would like to retire the old version of the function.
> -After having gone through the ABI deprecation announcement process, removal is
> -easy. Start by removing the symbol from the requisite version map file:
> -
> -.. code-block:: none
> -
> - DPDK_2.0 {
> - global:
> -
> - rte_acl_add_rules;
> - rte_acl_build;
> - rte_acl_classify;
> - rte_acl_classify_alg;
> - rte_acl_classify_scalar;
> - rte_acl_dump;
> - - rte_acl_create
> - rte_acl_find_existing;
> - rte_acl_free;
> - rte_acl_ipv4vlan_add_rules;
> - rte_acl_ipv4vlan_build;
> - rte_acl_list_dump;
> - rte_acl_reset;
> - rte_acl_reset_rules;
> - rte_acl_set_ctx_classify;
> -
> - local: *;
> - };
> -
> - DPDK_2.1 {
> - global:
> - rte_acl_create;
> - } DPDK_2.0;
> -
> -
> -Next remove the corresponding versioned export.
> -
> -.. code-block:: c
> -
> - -VERSION_SYMBOL(rte_acl_create, _v20, 2.0);
> -
> -
> -Note that the internal function definition could also be removed, but its used
> -in our example by the newer version _v21, so we leave it in place. This is a
> -coding style choice.
> -
> -Lastly, we need to bump the LIBABIVER number for this library in the Makefile to
> -indicate to applications doing dynamic linking that this is a later, and
> -possibly incompatible library version:
> -
> -.. code-block:: c
> -
> - -LIBABIVER := 1
> - +LIBABIVER := 2
> -
> -Deprecating an entire ABI version
> -_________________________________
> -
> -While removing a symbol from and ABI may be useful, it is often more practical
> -to remove an entire version node at once. If a version node completely
> -specifies an API, then removing part of it, typically makes it incomplete. In
> -those cases it is better to remove the entire node
> -
> -To do this, start by modifying the version map file, such that all symbols from
> -the node to be removed are merged into the next node in the map
> -
> -In the case of our map above, it would transform to look as follows
> -
> -.. code-block:: none
> -
> - DPDK_2.1 {
> - global:
> -
> - rte_acl_add_rules;
> - rte_acl_build;
> - rte_acl_classify;
> - rte_acl_classify_alg;
> - rte_acl_classify_scalar;
> - rte_acl_dump;
> - rte_acl_create
> - rte_acl_find_existing;
> - rte_acl_free;
> - rte_acl_ipv4vlan_add_rules;
> - rte_acl_ipv4vlan_build;
> - rte_acl_list_dump;
> - rte_acl_reset;
> - rte_acl_reset_rules;
> - rte_acl_set_ctx_classify;
> -
> - local: *;
> - };
> -
> -Then any uses of BIND_DEFAULT_SYMBOL that pointed to the old node should be
> -updated to point to the new version node in any header files for all affected
> -symbols.
> -
> -.. code-block:: c
> -
> - -BIND_DEFAULT_SYMBOL(rte_acl_create, _v20, 2.0);
> - +BIND_DEFAULT_SYMBOL(rte_acl_create, _v21, 2.1);
> -
> -Lastly, any VERSION_SYMBOL macros that point to the old version node should be
> -removed, taking care to keep, where need old code in place to support newer
> -versions of the symbol.
> -
> -
> -Running the ABI Validator
> --------------------------
> -
> -The ``devtools`` directory in the DPDK source tree contains a utility program,
> -``validate-abi.sh``, for validating the DPDK ABI based on the Linux `ABI
> -Compliance Checker
> -<http://ispras.linuxbase.org/index.php/ABI_compliance_checker>`_.
> -
> -This has a dependency on the ``abi-compliance-checker`` and ``and abi-dumper``
> -utilities which can be installed via a package manager. For example::
> -
> - sudo yum install abi-compliance-checker
> - sudo yum install abi-dumper
> -
> -The syntax of the ``validate-abi.sh`` utility is::
> -
> - ./devtools/validate-abi.sh <REV1> <REV2>
> -
> -Where ``REV1`` and ``REV2`` are valid gitrevisions(7)
> -https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html
> -on the local repo.
> -
> -For example::
> -
> - # Check between the previous and latest commit:
> - ./devtools/validate-abi.sh HEAD~1 HEAD
> -
> - # Check on a specific compilation target:
> - ./devtools/validate-abi.sh -t x86_64-native-linux-gcc HEAD~1 HEAD
> -
> - # Check between two tags:
> - ./devtools/validate-abi.sh v2.0.0 v2.1.0
> -
> - # Check between git master and local topic-branch "vhost-hacking":
> - ./devtools/validate-abi.sh master vhost-hacking
> -
> -After the validation script completes (it can take a while since it need to
> -compile both tags) it will create compatibility reports in the
> -``./abi-check/compat_report`` directory. Listed incompatibilities can be found
> -as follows::
> -
> - grep -lr Incompatible abi-check/compat_reports/
More information about the dev
mailing list