<div dir="ltr">Tested-by: Patrick Robb <<a href="mailto:probb@iol.unh.edu">probb@iol.unh.edu</a>></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 3, 2023 at 5:25 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Document how to configure and run DTS. Also add documentation related to new features: SUT setup and a brief test suite implementation cookbook. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- doc/guides/tools/dts.rst | 165 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 163 insertions(+), 2 deletions(-) diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst index daf54359ed..ebd6dceb6a 100644 --- a/doc/guides/tools/dts.rst +++ b/doc/guides/tools/dts.rst @@ -1,5 +1,5 @@ .. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2022 PANTHEON.tech s.r.o. + Copyright(c) 2022-2023 PANTHEON.tech s.r.o. DPDK Test Suite =============== @@ -56,7 +56,7 @@ DTS runtime environment or just plain DTS environment are used interchangeably. Setting up DTS environment --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ #. **Python Version** @@ -93,6 +93,167 @@ Setting up DTS environment poetry install poetry shell +#. **SSH Connection** + + DTS uses Python pexpect for SSH connections between DTS environment and the other hosts. + The pexpect implementation is a wrapper around the ssh command in the DTS environment. + This means it'll use the SSH agent providing the ssh command and its keys. + + +Setting up System Under Test +---------------------------- + +There are two areas that need to be set up on a System Under Test: + +#. **DPDK dependencies** + + DPDK will be built and run on the SUT. + Consult the Getting Started guides for the list of dependencies for each distribution. + +#. **Hardware dependencies** + + Any hardware DPDK uses needs a proper driver + and most OS distributions provide those, but the version may not be satisfactory. + It's up to each user to install the driver they're interested in testing. + The hardware also may also need firmware upgrades, which is also left at user discretion. + +#. **Hugepages** + + There are two ways to configure hugepages: + + * DTS configuration + + You may specify the optional hugepage configuration in the DTS config file. + If you do, DTS will take care of configuring hugepages, + overwriting your current SUT hugepage configuration. + + * System under test configuration + + It's possible to use the hugepage configuration already present on the SUT. + If you wish to do so, don't specify the hugepage configuration in the DTS config file. + + +Running DTS +----------- + +DTS needs to know which nodes to connect to and what hardware to use on those nodes. +Once that's configured, DTS needs a DPDK tarball and it's ready to run. + +Configuring DTS +~~~~~~~~~~~~~~~ + +DTS configuration is split into nodes and executions and build targets within executions. +By default, DTS will try to use the ``dts/conf.yaml`` config file, +which is a template that illustrates what can be configured in DTS: + + .. literalinclude:: ../../../dts/conf.yaml + :language: yaml + :start-at: executions: + + +The user must be root or any other user with prompt starting with ``#``. +The other fields are mostly self-explanatory +and documented in more detail in ``dts/framework/config/conf_yaml_schema.json``. + +DTS Execution +~~~~~~~~~~~~~ + +DTS is run with ``main.py`` located in the ``dts`` directory after entering Poetry shell:: + + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] + [-v VERBOSE] [-s SKIP_SETUP] [--tarball TARBALL] + [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] + [--re-run RE_RUN] + + Run DPDK test suites. All options may be specified with the environment variables provided in + brackets. Command line arguments have higher priority. + + options: + -h, --help show this help message and exit + --config-file CONFIG_FILE + [DTS_CFG_FILE] configuration file that describes the test cases, SUTs + and targets. (default: conf.yaml) + --output-dir OUTPUT_DIR, --output OUTPUT_DIR + [DTS_OUTPUT_DIR] Output directory where dts logs and results are + saved. (default: output) + -t TIMEOUT, --timeout TIMEOUT + [DTS_TIMEOUT] The default timeout for all DTS operations except for + compiling DPDK. (default: 15) + -v VERBOSE, --verbose VERBOSE + [DTS_VERBOSE] Set to 'Y' to enable verbose output, logging all + messages to the console. (default: N) + -s SKIP_SETUP, --skip-setup SKIP_SETUP + [DTS_SKIP_SETUP] Set to 'Y' to skip all setup steps on SUT and TG + nodes. (default: N) + --tarball TARBALL, --snapshot TARBALL + [DTS_DPDK_TARBALL] Path to DPDK source code tarball which will be + used in testing. (default: dpdk.tar.xz) + --compile-timeout COMPILE_TIMEOUT + [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200) + --test-cases TEST_CASES + [DTS_TESTCASES] Comma-separated list of test cases to execute. + Unknown test cases will be silently ignored. (default: ) + --re-run RE_RUN, --re_run RE_RUN + [DTS_RERUN] Re-run each test case the specified amount of times if a + test failure occurs (default: 0) + + +The brackets contain the names of environment variables that set the same thing. +The minimum DTS needs is a config file and a DPDK tarball. +You may pass those to DTS using the command line arguments or use the default paths. + + +DTS Results +~~~~~~~~~~~ + +Results are stored in the output dir by default +which be changed with the ``--output-dir`` command line argument. +The results contain basic statistics of passed/failed test cases and DPDK version. + + +How To Write a Test Suite +------------------------- + +All test suites inherit from ``TestSuite`` defined in ``dts/framework/test_suite.py``. +There are four types of methods that comprise a test suite: + +#. **Test cases** + + | Test cases are methods that start with a particular prefix. + | Functional test cases start with ``test_``, e.g. ``test_hello_world_single_core``. + | Performance test cases start with ``test_perf_``, e.g. ``test_perf_nic_single_core``. + | A test suite may have any number of functional and/or performance test cases. + However, these test cases must test the same feature, + following the rule of one feature = one test suite. + Test cases for one feature don't need to be grouped in just one test suite, though. + If the feature requires many testing scenarios to cover, + the test cases would be better off spread over multiple test suites + so that each test suite doesn't take too long to execute. + +#. **Setup and Teardown methods** + + | There are setup and teardown methods for the whole test suite and each individual test case. + | Methods ``set_up_suite`` and ``tear_down_suite`` will be executed + before any and after all test cases have been executed, respectively. + | Methods ``set_up_test_case`` and ``tear_down_test_case`` will be executed + before and after each test case, respectively. + | These methods don't need to be implemented if there's no need for them in a test suite. + In that case, nothing will happen when they're is executed. + +#. **Test case verification** + + Test case verification should be done with the ``verify`` method, which records the result. + The method should be called at the end of each test case. + +#. **Other methods** + + Of course, all test suite code should adhere to coding standards. + Only the above methods will be treated specially and any other methods may be defined + (which should be mostly private methods needed by each particular test suite). + Any specific features (such as NIC configuration) required by a test suite + should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses) + and used by the test suite via the ``sut_node`` field. + DTS Developer Tools ------------------- -- 2.30.2 </blockquote></div> <div> </div>-- <div dir="ltr" class="gmail_signature"><div dir="ltr">Patrick RobbTechnical Service ManagerUNH InterOperability Laboratory21 Madbury Rd, Suite 100, Durham, NH 03824<a href="http://www.iol.unh.edu/" style="color:rgb(17,85,204)" target="_blank">www.iol.unh.edu</a> <img src="https://lh4.googleusercontent.com/7sTY8VswXadak_YT0J13osh5ockNVRX2BuYaRsKoTTpkpilBokA0WlocYHLB4q7XUgXNHka6-ns47S8R_am0sOt7MYQQ1ILQ3S-P5aezsrjp3-IsJMmMrErHWmTARNgZhpAx06n2" width="150" height="37" style="border: none;"></div></div>