[PATCH 1/1] tools: check for pending test status when parsing emails

[Aaron Conole]

Patrick Robb <probb at iol.unh.edu> writes:

> On Mon, May 20, 2024 at 3:03 PM Thomas Monjalon <thomas at monjalon.net> wrote:
>>
>> 17/05/2024 21:22, Patrick Robb:
>> > Signed-off-by: Patrick Robb <probb at iol.unh.edu>
>>
>> Please could you explain what it is doing?
>> Having a workflow understanding would be nice.

+1.  It would usually be used in the commit message to show the
rationale for the change.  Here's a suggestion for an even shorter
distillation from your message below:

 Today, the community CI infrastructure only uses post-result reporting,
 such as "SUCCESS", "FAILED", and "WARNING".  These results get reported
 only after a test finishes.  This creates some confusion about whether a
 test might have been started for the series in question.  It isn't easy
 to tell at-a-glance which tests are currently running for a given patch
 or series.

 This patch aims to introduce support for a "PENDING" state in the CI
 infrastructure.  This allows labs to indicate which tests have started
 and are awaiting results.  That means test writers should now send a
 "PENDING" status for tests as they start, and then update with a
 post-test result after.  With this change, understanding which tests ran
 at-a-glance is something we can achieve.

 This change should have no affect on the actual tests being run.

Maybe a v2 with something like this as the commit message?  WDYT?

> Yes good idea.
>
> For context, pending is already a supported check state in patchwork
> server: https://patchwork.readthedocs.io/en/latest/usage/overview/#checks
>
> 1. DPDK patch is submitted. Patch is acquired by UNH Lab.
> 2. UNH Lab triggers some testrun pipelines in our CI system (jenkins).
> The first action the pipeline takes is to create in our database a
> test result record for each testrun, setting the status to PENDING. It
> is important to note that one patchwork context, Like
> "iol-compile-amd64-testing," may consist of many individual testruns,
> each for different distros, hardware, environment etc.
> 3. When each testrun completes, it will send a report to Patchwork
> with the new result (pass or fail). When it does this it will update
> the context's results table, changing the environment's result from
> pending to pass/fail. So, when the first report comes in for, say,
> context "iol-compile-amd64-testing," you would see 1 pass/fail, 12
> pending, or similar. Then, as subsequent testruns complete, and report
> their results, the updated table comes with the new report. The
> overall context result (the _Testing {PASS/FAIL/PENDING}_ at the top
> of the test report email) is determined in the manner you might
> expect, i.e. if there is at least one testrun fail result, overall
> context is fail, else if there is at least one pending result, overall
> context is pending, else if all results are passing, overall result is
> passing. As an example, when testing is nearly complete, the top of
> the report email may look like this:
>
> _Testing PENDING_
>
> Branch: tags/v22.11
>
> a409653a123bf105970a25c594711a3cdc44d139 --> testing pass
>
> Test environment and result as below:
>
> +------------------------------------+-----------------------------------------------------+
> |            Environment             |       dpdk_meson_compile      |
> +====================================+====================+
> | Ubuntu 20.04 ARM SVE                          | PASS               |
> +------------------------------------+--------------------+
> | Debian 12 with MUSDK                           | PENDING        |
> +------------------------------------+--------------------+
> | Fedora 37 (ARM)                                     | PASS               |
> +------------------------------------+--------------------+
> | Ubuntu 20.04 (ARM)                                | PASS               |
> +------------------------------------+--------------------+
> | Fedora 38 (ARM)                                     | PASS               |
> +------------------------------------+--------------------+
> | Fedora 39 (ARM)                                     | PENDING        |
> +------------------------------------+--------------------+
> | Debian 12 (arm)                                       | PASS               |
> +------------------------------------+--------------------+
> | CentOS Stream 9 (ARM)                         | PASS               |
> +------------------------------------+--------------------+
> | Debian 11 (Buster) (ARM)                        | PASS               |
> +------------------------------------+--------------------+
> | Ubuntu 20.04 ARM GCC Cross Compile | PASS               |
> +------------------------------------+--------------------+
>
>
> 4. Eventually, all testruns are complete for a patchwork context, and
> the table switches from pending to pass or fail.
>
> This does not slow the delivery of results, nor does it increase the
> number of test report emails sent. We still send only 1 email per
> testrun.
>
> This way it is plainly visible to the user when all testing is
> complete, and it also flags for the submitter and for CI people if
> some infra failure prevents a testrun from completing, or from a
> result being properly emailed, etc. The idea is to provide more
> complete status updates and check against infra fails better, but
> without any adverse effects in user experience or load on the email
> server.
>
>>
>> > --- a/tools/update-pw.sh
>> > +++ b/tools/update-pw.sh
>> > @@ -49,6 +49,7 @@ case $status in
>> >       'SUCCESS') pwstatus='success' ;;
>> >       'WARNING') pwstatus='warning' ;;
>> >       'FAILURE') pwstatus='fail' ;;
>> > +     'PENDING') pwstatus='pending' ;;
>>
>>
>>