<div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 27, 2025 at 11:37 AM Dean Marx <<a href="mailto:dmarx@iol.unh.edu" target="_blank">dmarx@iol.unh.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Modify dts.rst to exclude redundant/outdated information about the project,<br> and add new information regarding setup and framework design.<br> <br> Signed-off-by: Dean Marx <<a href="mailto:dmarx@iol.unh.edu" target="_blank">dmarx@iol.unh.edu</a>><br> ---<br> doc/guides/tools/dts.rst | 310 +++++++++++++--------------------------<br> 1 file changed, 99 insertions(+), 211 deletions(-)<br> <br> diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst<br> index fcc6d22036..0aa6663b9f 100644<br> --- a/doc/guides/tools/dts.rst<br> +++ b/doc/guides/tools/dts.rst<br> @@ -1,6 +1,7 @@<br> .. SPDX-License-Identifier: BSD-3-Clause<br> Copyright(c) 2022-2023 PANTHEON.tech s.r.o.<br> Copyright(c) 2024 Arm Limited<br> + Copyright(c) 2025 University of New Hampshire<br> <br> DPDK Test Suite<br> ===============<br> @@ -20,31 +21,18 @@ DTS runtime environment<br> <br> DTS runtime environment node<br> A node where at least one DTS runtime environment is present.<br> - This is the node where we run DTS and from which DTS connects to other nodes.<br></blockquote><div><br></div><div>Leave this in.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> System under test<br> - An SUT is the combination of DPDK and the hardware we're testing<br> - in conjunction with DPDK (NICs, crypto and other devices).<br> + Node with DPDK and relevant hardware (NICs, crypto, etc.).<br></blockquote><div><br></div><div>Maybe change to "The system which runs a DPDK application on relevant hardware (NIC, accelerator cards, etc) and from which the DPDK behavior is observed for tests."</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> System under test node<br> A node where at least one SUT is present.<br> <br> Traffic generator<br> - A TG is either software or hardware capable of sending packets.<br> + Node that sends traffic; can be hardware or software-based.<br></blockquote><div><br></div><div>"Node that sends traffic to the SUT;"</div><div> </div><div>Sorry for being so particular. :)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> Traffic generator node<br> A node where at least one TG is present.<br> - In case of hardware traffic generators, the TG and the node are literally the same.<br> -<br> -<br> -In most cases, interchangeably referring to a runtime environment, SUT, TG or the node<br> -they're running on (e.g. using SUT and SUT node interchangeably) doesn't cause confusion.<br> -There could theoretically be more than of these running on the same node and in that case<br> -it's useful to have stricter definitions.<br> -An example would be two different traffic generators (such as Trex and Scapy)<br> -running on the same node.<br> -A different example would be a node containing both a DTS runtime environment<br> -and a traffic generator, in which case it's both a DTS runtime environment node and a TG node.<br> <br> <br> DTS Environment<br> @@ -195,12 +183,28 @@ These need to be set up on a Traffic Generator Node:<br> Running DTS<br> -----------<br> <br> -DTS needs to know which nodes to connect to and what hardware to use on those nodes.<br> -Once that's configured, either a DPDK source code tarball or tree folder<br> -need to be supplied whether these are on your DTS host machine or the SUT node.<br> -DTS can accept a pre-compiled build placed in a subdirectory,<br> -or it will compile DPDK on the SUT node,<br> -and then run the tests with the newly built binaries.<br> +To run DTS, use ``main.py`` with Poetry:<br> +<br> +.. code-block:: console<br> +<br> + ```shell<br> + docker build --target dev -t dpdk-dts .<br> + docker run -v $(pwd)/..:/dpdk -v /home/dtsuser/.ssh:/root/.ssh:ro -it dpdk-dts bash<br> + $ poetry install<br> + $ poetry run ./main.py<br> + ```<br> +<br> +Common options include:<br> +<br> +- ``--output-dir``: Custom output location.<br> +- ``--remote-source``: Use sources stored on the SUT.<br> +- ``--tarball``: Specify the tarball to be tested.<br> +<br> +For a full list:<br> +<br> +.. code-block:: console<br> +<br> + poetry run ./main.py --help<br></blockquote><div><br></div><div>I think we should keep the full list of flags here instead of removing it for this subset. It's a bit of a maintenance burden and it make the file longer but it's important info. I think it's good to present it here even if it is only "a --help away."</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> <br> Configuring DTS<br> @@ -220,71 +224,6 @@ The user must have :ref:`administrator privileges <sut_admin_user>`<br> which don't require password authentication.<br> <br> <br> -DTS Execution<br> -~~~~~~~~~~~~~<br> -<br> -DTS is run with ``main.py`` located in the ``dts`` directory after entering Poetry shell:<br> -<br> -.. code-block:: console<br> -<br> - (dts-py3.10) $ ./main.py --help<br> - usage: main.py [-h] [--test-run-config-file FILE_PATH] [--nodes-config-file FILE_PATH] [--tests-config-file FILE_PATH]<br> - [--output-dir DIR_PATH] [-t SECONDS] [-v] [--dpdk-tree DIR_PATH | --tarball FILE_PATH] [--remote-source]<br> - [--precompiled-build-dir DIR_NAME] [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]]<br> - [--re-run N_TIMES] [--random-seed NUMBER]<br> -<br> - Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher<br> - priority.<br> -<br> - options:<br> - -h, --help show this help message and exit<br> - --test-run-config-file FILE_PATH<br> - [DTS_TEST_RUN_CFG_FILE] The configuration file that describes the test cases and DPDK build options. (default: test-run.conf.yaml)<br> - --nodes-config-file FILE_PATH<br> - [DTS_NODES_CFG_FILE] The configuration file that describes the SUT and TG nodes. (default: nodes.conf.yaml)<br> - --tests-config-file FILE_PATH<br> - [DTS_TESTS_CFG_FILE] Configuration file used to override variable values inside specific test suites. (default: None)<br> - --output-dir DIR_PATH, --output DIR_PATH<br> - [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)<br> - -t SECONDS, --timeout SECONDS<br> - [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)<br> - -v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)<br> - --compile-timeout SECONDS<br> - [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)<br> - --test-suite TEST_SUITE [TEST_CASES ...]<br> - [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and<br> - the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites<br> - in the environment variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite<br> - suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case<br> - ... | DTS_TEST_SUITES='suite, suite case, ...' (default: [])<br> - --re-run N_TIMES, --re_run N_TIMES<br> - [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)<br> - --random-seed NUMBER [DTS_RANDOM_SEED] The seed to use with the pseudo-random generator. If not specified, the configuration value is<br> - used instead. If that's also not specified, a random seed is generated. (default: None)<br> -<br> - DPDK Build Options:<br> - Arguments in this group (and subgroup) will be applied to a DPDKLocation when the DPDK tree, tarball or revision will be provided,<br> - other arguments like remote source and build dir are optional. A DPDKLocation from settings are used instead of from config if<br> - construct successful.<br> -<br> - --dpdk-tree DIR_PATH [DTS_DPDK_TREE] The path to the DPDK source tree directory to test. Cannot be used in conjunction with --tarball.<br> - (default: None)<br> - --tarball FILE_PATH, --snapshot FILE_PATH<br> - [DTS_DPDK_TARBALL] The path to the DPDK source tarball to test. DPDK must be contained in a folder with the same<br> - name as the tarball file. Cannot be used in conjunction with --dpdk-tree. (default: None)<br> - --remote-source [DTS_REMOTE_SOURCE] Set this option if either the DPDK source tree or tarball to be used are located on the SUT<br> - node. Can only be used with --dpdk-tree or --tarball. (default: False)<br> - --precompiled-build-dir DIR_NAME<br> - [DTS_PRECOMPILED_BUILD_DIR] Define the subdirectory under the DPDK tree root directory or tarball where the pre-<br> - compiled binaries are located. (default: None)<br> -<br> -<br> -The brackets contain the names of environment variables that set the same thing.<br> -The minimum DTS needs is a config file and a pre-built DPDK<br> -or DPDK sources location which can be specified in said config file<br> -or on the command line or environment variables.<br> -<br> -<br> DTS Results<br> ~~~~~~~~~~~<br> <br> @@ -308,140 +247,89 @@ Adding test cases may require adding code to the framework as well.<br> Framework Coding Guidelines<br> ~~~~~~~~~~~~~~~~~~~~~~~~~~~<br> <br> -When adding code to the DTS framework, pay attention to the rest of the code<br> -and try not to divert much from it.<br> -The :ref:`DTS developer tools <dts_dev_tools>` will issue warnings<br> -when some of the basics are not met.<br> -You should also build the :ref:`API documentation <building_api_docs>`<br> -to address any issues found during the build.<br> -<br> -The API documentation, which is a helpful reference when developing, may be accessed<br> -in the code directly or generated with the :ref:`API docs build steps <building_api_docs>`.<br> -When adding new files or modifying the directory structure,<br> -the corresponding changes must be made to DTS API doc sources in ``doc/api/dts``.<br> -<br> -Speaking of which, the code must be properly documented with docstrings.<br> -The style must conform to the `Google style<br> -<<a href="https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings" rel="noreferrer" target="_blank">https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings</a>>`_.<br> -See an example of the style `here<br> -<<a href="https://www.sphinx-doc.org/en/master/usage/extensions/example_google.html" rel="noreferrer" target="_blank">https://www.sphinx-doc.org/en/master/usage/extensions/example_google.html</a>>`_.<br> -For cases which are not covered by the Google style, refer to `PEP 257<br> -<<a href="https://peps.python.org/pep-0257/" rel="noreferrer" target="_blank">https://peps.python.org/pep-0257/</a>>`_.<br> -There are some cases which are not covered by the two style guides,<br> -where we deviate or where some additional clarification is helpful:<br> -<br> - * The ``__init__()`` methods of classes are documented separately<br> - from the docstring of the class itself.<br> - * The docstrings of implemented abstract methods should refer to the superclass's definition<br> - if there's no deviation.<br> - * Instance variables/attributes should be documented in the docstring of the class<br> - in the ``Attributes:`` section.<br> - * The ``dataclass.dataclass`` decorator changes how the attributes are processed.<br> - The dataclass attributes which result in instance variables/attributes<br> - should also be recorded in the ``Attributes:`` section.<br> - * Class variables/attributes and Pydantic model fields, on the other hand,<br> - should be documented with ``#:`` above the type annotated line.<br> - The description may be omitted if the meaning is obvious.<br> - * The ``Enum`` and ``TypedDict`` also process the attributes in particular ways<br> - and should be documented with ``#:`` as well.<br> - This is mainly so that the autogenerated documentation contains the assigned value.<br> - * When referencing a parameter of a function or a method in their docstring,<br> - don't use any articles and put the parameter into single backticks.<br> - This mimics the style of `Python's documentation <<a href="https://docs.python.org/3/index.html" rel="noreferrer" target="_blank">https://docs.python.org/3/index.html</a>>`_.<br> - * When specifying a value, use double backticks::<br> -<br> - def foo(greet: bool) -> None:<br> - """Demonstration of single and double backticks.<br> -<br> - `greet` controls whether ``Hello World`` is printed.<br> -<br> - Args:<br> - greet: Whether to print the ``Hello World`` message.<br> - """<br> - if greet:<br> - print(f"Hello World")<br> -<br> - * The docstring maximum line length is the same as the code maximum line length.<br> -<br> -<br> -How To Write a Test Suite<br> --------------------------<br> -<br> -All test suites inherit from ``TestSuite`` defined in ``dts/framework/test_suite.py``.<br></blockquote><div><br></div><div>"All test suites are a class which inherits from"</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> -There are four types of methods that comprise a test suite:<br> -<br> -#. **Test cases**<br> -<br> - | Test cases are methods that start with a particular prefix.<br> - | Functional test cases start with ``test_``, e.g. ``test_hello_world_single_core``.</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> - | Performance test cases start with ``test_perf_``, e.g. ``test_perf_nic_single_core``.<br></blockquote><div><br></div><div><div>Now decorator based.</div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> - | A test suite may have any number of functional and/or performance test cases.<br> - However, these test cases must test the same feature,<br> - following the rule of one feature = one test suite.<br> - Test cases for one feature don't need to be grouped in just one test suite, though.<br> - If the feature requires many testing scenarios to cover,<br> - the test cases would be better off spread over multiple test suites<br> - so that each test suite doesn't take too long to execute.<br> -<br> -#. **Setup and Teardown methods**<br> -<br> - | There are setup and teardown methods for the whole test suite and each individual test case.<br> - | Methods ``set_up_suite`` and ``tear_down_suite`` will be executed<br> - before any and after all test cases have been executed, respectively.<br> - | Methods ``set_up_test_case`` and ``tear_down_test_case`` will be executed<br> - before and after each test case, respectively.<br> - | These methods don't need to be implemented if there's no need for them in a test suite.<br> - In that case, nothing will happen when they are executed.<br> -<br> -#. **Configuration, traffic and other logic**<br> -<br> - The ``TestSuite`` class contains a variety of methods for anything that<br> - a test suite setup, a teardown, or a test case may need to do.<br> -<br> - The test suites also frequently use a DPDK app, such as testpmd, in interactive mode<br> - and use the interactive shell instances directly.<br> -<br> - These are the two main ways to call the framework logic in test suites.<br> - If there's any functionality or logic missing from the framework,<br> - it should be implemented so that the test suites can use one of these two ways.<br> -<br> -#. **Test case verification**<br> -<br> - Test case verification should be done with the ``verify`` method, which records the result.<br> - The method should be called at the end of each test case.<br> -<br> -#. **Other methods**<br></blockquote><div><br></div><div>I see that some of the content under "Other methods" is now false and should be removed - thanks for doing so. However, I do think there was a lot of good within the original "Test Cases," "Setup and Teardown Methods," and "Configuration, traffic and other logic" which has been removed. For this one I prefer if we just sit down and hash it out in person when you're in next week.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> -<br> - Of course, all test suite code should adhere to coding standards.<br> - Only the above methods will be treated specially and any other methods may be defined<br> - (which should be mostly private methods needed by each particular test suite).<br> - Any specific features (such as NIC configuration) required by a test suite<br> - should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)<br> - and used by the test suite via the ``sut_node`` field.<br> +When contributing code to the DTS framework, follow existing conventions to ensure consistency.<br> +The :ref:`DTS developer tools <dts_dev_tools>` will flag basic issues.<br> +Also, be sure to :ref:`build the API documentation <building_api_docs>` to catch any problems during the build.<br> <br> +The API documentation is a helpful reference during development.<br> +It can be viewed in the code directly or generated using the :ref:`API docs build steps <building_api_docs>`.<br> +If you add new files or change the directory structure, update the corresponding sources in ``doc/api/dts``.<br> <br> -.. _dts_dev_tools:<br> +Code must be documented with docstrings that follow the<br> +`Google style <<a href="https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings" rel="noreferrer" target="_blank">https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings</a>>`_.<br> +Additional references:<br> <br> -DTS Developer Tools<br> --------------------<br> +* `Sphinx Google style example <<a href="https://www.sphinx-doc.org/en/master/usage/extensions/example_google.html" rel="noreferrer" target="_blank">https://www.sphinx-doc.org/en/master/usage/extensions/example_google.html</a>>`_<br> +* `PEP 257 <<a href="https://peps.python.org/pep-0257/" rel="noreferrer" target="_blank">https://peps.python.org/pep-0257/</a>>`_<br> +<br> +Docstring and Attribute Guidelines<br> <br> -There are two tools used in DTS to help with code checking, style and formatting:<br> +* Document ``__init__()`` separately from the class docstring.<br> +* If an abstract method simply implements a superclass definition without changes, refer to that superclass in the docstring.<br> +* Document instance variables in the class docstring under an ``Attributes:`` section.<br> +* For ``@dataclass`` classes, document instance-level attributes in ``Attributes:``, as they are generated from the class fields.<br> +* Document class variables and Pydantic fields using ``#:``, <br> + placed above the type-annotated line. Descriptions may be omitted if the meaning is clear.<br> +* Apply ``#:`` to ``Enum`` and ``TypedDict`` fields as well, so that autogenerated documentation includes their values.<br> +* When referring to a parameter in a docstring, omit articles and enclose the parameter in single backticks (e.g., `` `param` ``),<br> + consistent with the `Python documentation style <<a href="https://docs.python.org/3/index.html" rel="noreferrer" target="_blank">https://docs.python.org/3/index.html</a>>`_.<br> +* Use double backticks (````value````) for literal values.<br> <br> -* `ruff <<a href="https://astral.sh/ruff/" rel="noreferrer" target="_blank">https://astral.sh/ruff/</a>>`_<br> +Example::<br> <br> - An extremely fast all-in-one linting and formatting solution,<br> - which covers most if not all the major rules such as:<br> - pylama, flake8, pylint etc.<br> - Its built-in formatter is also Black-compatible<br> - and is able to sort imports automatically like isort would.<br> + def foo(greet: bool) -> None:<br> + """Demonstrates single vs. double backticks.<br> <br> -* `mypy <<a href="https://github.com/python/mypy" rel="noreferrer" target="_blank">https://github.com/python/mypy</a>>`_<br> + `greet` controls whether ``Hello World`` is printed.<br> <br> - Enables static typing for Python, exploiting the type hints in the source code.<br> + Args:<br> + greet: Whether to print the ``Hello World`` message.<br> + """<br> + if greet:<br> + print("Hello World")<br> +<br> +The maximum line length for docstrings must match that of the code.<br> +<br> +<br> +Creating a Test Suite<br> +---------------------<br> +<br> +All test suites inherit from ``TestSuite`` in ``dts/framework/test_suite.py``. A typical suite contains:<br> +<br> +- Test Cases <br> + - Functional: start with ``test_``, e.g., ``test_basic_link`` <br> + - Performance: start with ``test_perf_``, e.g., ``test_perf_throughput`` </blockquote><div><br></div><div>I realize you go on to describe the decorators after this, but I think the test_func and test_perf naming convention is no longer required. Example: </div><div><br></div><div> @requires(NicCapability.FLOW_CTRL)<br> @func_test<br> def flow_ctrl_port_configuration_persistence(self) -> None:<br> """Flow control port configuration persistency test.<br><br> Steps:<br> For each port enable flow control for RX and TX individually.<br> Verify:<br> The configuration persists after the port is restarted.<br> """</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> + - Import the ``func_test`` and/or ``perf_test`` decorators from ``TestSuite`` and add them above each test case method,<br> + e.g., ``@func_test`` followed by ``test_basic_link``<br> +<br> +- Setup/Teardown Hooks<br> + - Suite-level: ``set_up_suite()``, ``tear_down_suite()``<br> + - Case-level: ``set_up_test_case()``, ``tear_down_test_case()``<br> +<br> +- Verification <br> + Use ``self.verify(condition, message)`` to record test results.<br></blockquote><div><br></div><div>I think the important part of "verify" to explain to people is that you are setting the testcase assertion condition, not the recording aspect.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> +<br> +- Support Methods<br> + Helper logic (e.g., traffic handling, config) should be in private methods or delegated to ``sut_node``.<br></blockquote><div><br></div><div>I realize this is just a rewording that crept in, but this is wrong for a couple reasons:</div><div><br></div><div>1. We no longer import node/sutnode when writing testsuites. Node is purely framework code now, and is not exposed to testsuites.</div><div>2. sut_node (and tg_node) no longer exists.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> +<br> +<br> +.. _dts_dev_tools:<br> +<br> +Developer Tools<br> +---------------<br> +<br> +- ruff <br> + - Linter and formatter (replaces flake8, pylint, isort, etc.)<br> + - Compatible with Black<br> +<br> +- mypy <br> + - Performs static type checking<br> +<br> +Run checks using:<br> +<br> +.. code-block:: console<br> <br> -These two tools are all used in ``devtools/dts-check-format.sh``,<br> -the DTS code check and format script.<br> -Refer to the script for usage: ``devtools/dts-check-format.sh -h``.<br> + devtools/dts-check-format.sh<br> <br> <br> .. _building_api_docs:<br> -- <br> 2.49.0<br> <br></blockquote><div><br></div><div>Lots of improvements overall - keep up the good work! A productive Summer ahead. </div><div><br></div><div>Reviewed-by: Patrick Robb <<a href="mailto:probb@iol.unh.edu">probb@iol.unh.edu</a>></div><div> </div></div></div> </div>