[dpdk-web] [PATCH] add general testing guidelines

Thomas Monjalon thomas at monjalon.net
Tue Mar 10 23:11:30 CET 2020

The community lab page is replaced with a more general page about testing,
where the community lab is part of the big picture. It includes:
	- testing goals
	- CI workflow
	- tools

Signed-off-by: Thomas Monjalon <thomas at monjalon.net>
 content/lab/_index.md                |  21 ----
 content/testing/_index.md            | 142 +++++++++++++++++++++++++++
 static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes
 3 files changed, 142 insertions(+), 21 deletions(-)
 delete mode 100644 content/lab/_index.md
 create mode 100644 content/testing/_index.md
 delete mode 100644 static/img/community-lab-diagram.png

diff --git a/content/lab/_index.md b/content/lab/_index.md
deleted file mode 100644
index 913c86f..0000000
--- a/content/lab/_index.md
+++ /dev/null
@@ -1,21 +0,0 @@
-title = "Community Lab"
-weight = "8"
-{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{< /button >}}
-## What it's for
-- Automated performance regression testing of new patches.
-  - When a new patch is received, automatically test
-    and determine if it degrades performance or causes other issues
-    on one or more architectures.
-  - Provides input to maintainers on whether to accept patches or request rework.
-- The Dashboard currently provides performance deltas for each patch
-  compared to the expected numbers of each architecture.
-- The scope may expand in future:
-  test DPDK-enabled applications, use lab for demos, etc.
-![DPDK Community Lab](/img/community-lab-diagram.png)
diff --git a/content/testing/_index.md b/content/testing/_index.md
new file mode 100644
index 0000000..30da214
--- /dev/null
+++ b/content/testing/_index.md
@@ -0,0 +1,142 @@
+title = "Testing"
+weight = "8"
+As a community project, DPDK CI is distributed and welcome any contribution.
+There are various ways of contributing to the CI effort:
+- suggest improvement
+- write a new test
+- improve infrastucture scripts
+- run a lab
+A specific effort is done
+for the [community lab](//lab.dpdk.org/results/dashboard/about)
+hosted at [UNH-IOL](//www.iol.unh.edu).
+## CI Testing Goals
+### General Properties
+Ideally, the CI infrastructure should have these properties:
+- reliable
+- neutral
+- transparent visibility
+- coverage summary
+- trending graphs
+- reports to author and
+[test-report at dpdk.org](//inbox.dpdk.org/test-report/)
+- easy to add a test
+- easy to reproduce a test locally
+### Inputs
+There are three kinds of inputs to be tested:
+- per-patch testing integrated
+with [patchwork](//patches.dpdk.org/project/dpdk/list/)
+- per-commit testing after merges
+in mainline or dpdk-next- [repositories](//git.dpdk.org/)
+- per-release testing
+including [release candidates](//git.dpdk.org/dpdk/refs/)
+of all [branches](//git.dpdk.org/dpdk-stable/refs/)
+### Categories
+The tests should cover these categories:
+- compilation
+- static analysis
+- functional
+- performance
+- interoperability
+- portability
+- compatibility
+## Workflow
+### Patch Testing
+When a patch is submitted, it is received
+via the mailing list [dpdk-dev](//inbox.dpdk.org/dev/).
+For simple tests,
+like [checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh),
+it can be run directly on email receiving.
+For most tests, the patch must be applied
+on the [right tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py)
+after its series dependencies.
+Dealing with series and dependencies is easier
+via [patchwork database request](//patches.dpdk.org/api/).
+That's why the common workflow is
+to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw).
+The patches are saved and organized
+in [patchwork](//patches.dpdk.org/project/dpdk/list/)
+after being distributed by the mailing list.
+Once the patch is ready, the automatic tests can run.
+When a group of tests is finished, the author should be notified of any error,
+and a public report must be sent to
+[test-report at dpdk.org](//inbox.dpdk.org/test-report/).
+The reports are
+with a specific syntax for automatic parsing.
+When such report email is received, the bot ci at dpdk.org will
+[update patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh),
+referencing the report archive and incrementing one of the counters
+(success, warning or failure).
+A maintainer should wait the end of the tests (should be less than a day),
+and check the results before accepting a patch.
+### Mainline Monitoring
+The repository mirror on [GitHub](//github.com/DPDK/dpdk)
+is preferred for tests integration:
+- If polling for updates, better to load the GitHub mirror.
+- Some services like [Travis](//travis-ci.com/DPDK/)
+can be hooked in GitHub.
+### Release Validation
+When a new release is available, especially for release candidates,
+some validation reports are requested to the community members.
+Such release validations are sent as replies to the release announcement.
+## Tools
+### Infrastructure
+The testing infrastructure scripts are shared in
+the [dpdk-ci git repository](//git.dpdk.org/tools/dpdk-ci/).
+The website dashboard and API of the community lab is a Django project shared
+in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-site/).
+The testing infrastructure work is coordinated in three complementary forums:
+- [dpdk-ci mailing list](//inbox.dpdk.org/ci/)
+- [bugzilla tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20scripts&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpriority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_format=advanced&order=priority&list_id=1987)
+- bi-weekly meeting
+### Known Test Frameworks and Tools
+- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/)
+- [DPDK apps](//git.dpdk.org/dpdk/tree/app/)
+- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/)
+- [VSPERF](//git.opnfv.org/vswitchperf/about/)
+- [OvS PVP perf](//github.com/chaudron/ovs_perf/)
+- [downstream projects](//www.dpdk.org/ecosystem/#projects)
diff --git a/static/img/community-lab-diagram.png b/static/img/community-lab-diagram.png
deleted file mode 100644
index a32109a..0000000
Binary files a/static/img/community-lab-diagram.png and /dev/null differ

More information about the web mailing list