Email Based Re-Testing Framework

[Aaron Conole]

Patrick Robb <probb at iol.unh.edu> writes:

>  Also it can be useful to run daily sub-tree testing by request, if possible.
>
> That wouldn't be too difficult. I'll make a ticket for this. Although, for testing on the daily sub-trees, since that's
> UNH-IOL specific, that wouldn't necessarily have to be done via an email based testing request framework. We
> could also just add a button to our dashboard which triggers a sub-tree ci run. That would help keep narrow
> the scope of what the email based retesting framework is for. But, both email or a dashboard button would
> both work. 

We had discussed this long ago - including agreeing on a format, IIRC.

See the thread starting here:
  https://mails.dpdk.org/archives/ci/2021-May/001189.html

The idea was to have a line like:

Recheck-request: <test names>

where <test names> was the tests in the check labels.  In fact, what
started the discussion was a patch for the pw-ci scripts that
implemented part of it.

I don't see how to make your proposal as easily parsed.

WDYT?  Can you re-read that thread and come up with comments?

> On Tue, Jun 6, 2023 at 1:53 PM Ferruh Yigit <ferruh.yigit at amd.com> wrote:
>
>  On 6/6/2023 5:56 PM, Patrick Robb wrote:
>  > Hello all,
>  > 
>  > I'd like to revive the conversation about a request from the community
>  > for an email based re-testing framework. The idea is that using one
>  > standardized format, dpdk developers could email the test-report mailing
>  > list, requesting a rerun on their patch series for "X" set of tests at
>  > "Y" lab. I think that since patchwork testing labels (ie.
>  > iol-broadcom-Performance, github-robot: build, loongarch-compilation)
>  > are already visible on patch pages on patchwork, those labels are the
>  > most reasonable ones to expect developers to use when requesting a
>  > re-test. We probably wouldn't want to get any more general than that,
>  > like, say, rerunning all CI testing for a specific patch series at a
>  > specific lab, since it would result in a significant amount of "wasted"
>  > testing capacity.
>  > 
>  > The standard email format those of us at the Community Lab are thinking
>  > of is like below. Developers would request retests by emailing the
>  > test-report mailing list with email bodies like:
>  > 
>  > [RETEST UNH-IOL]
>  > iol-abi-testing
>  > iol-broadcom-Performance
>  > 
>  > [RETEST Intel]
>  > intel-Functional
>  > 
>  > [RETEST Loongson]
>  > loongarch-compilation
>  > 
>  > [RETEST GHA]
>  > github-robot: build
>  > 
>  > From there, it would be up to the various labs to poll the test-report
>  > mailing list archive (or use a similar method) to check for such
>  > requests, and trigger a CI testing rerun based on the labels provided in
>  > the re-test email. If there is interest from other labs, UNH might also
>  > be able to host the entire set of re-test requests, allowing other labs
>  > to poll a curated list hosted by UNH. One simple approach would be for
>  > labs to download all emails sent to test-report and parse with regex to
>  > determine the re-test list for their specific lab. But, if anyone has
>  > any better ideas for aggregating the emails to be parsed, suggestions
>  > are welcome! If this approach sounds reasonable to everyone, we could
>  > determine a timeline by which labs would implement the functionality
>  > needed to trigger re-tests. Or, we can just add re-testing for various
>  > labs if/when they add this functionality - whatever is better. Happy to
>  > discuss at the CI meeting on Thursday.
>  > 
>
>  +1 to re-testing framework.
>
>  Also it can be useful to run daily sub-tree testing by request, if possible.