Email Based Re-Testing Framework

[Patrick Robb]

>
> What would be the purpose of [RETEST UNH-IOL]?
>
Agreed, this is redundant, provided we will be using the labels/contexts
used on patchwork. That seems to be the idea behind Aaron's proposed format
and I think we should move to use his format since it previously reached
some consensus, and appears to be easy to parse.


> We need to specify the patchwork identifier of the patch.
>
> We could make a script similar to the checkpatch run on dpdk.org:
> https://git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh
> The easiest way to run it is to make the script as the receiver of the
> mail.
> If the lab can receive the mails from the mailing list,
> then just need to filter the retest requests for its own lab.

Yes, I think this is reasonable. I don't think this process is likely to
change much, and if we can provide a script to live on the dpdk-ci repo
which checks for retest requests, we can reasonably expect the labs to
separately set up an environment to handle running that script and
triggering their re-tests. Thanks Thomas.



On Wed, Jun 7, 2023 at 8:53 AM Aaron Conole <aconole at redhat.com> wrote:

> Patrick Robb <probb at iol.unh.edu> writes:
>
> >  Also it can be useful to run daily sub-tree testing by request, if
> possible.
> >
> > That wouldn't be too difficult. I'll make a ticket for this. Although,
> for testing on the daily sub-trees, since that's
> > UNH-IOL specific, that wouldn't necessarily have to be done via an email
> based testing request framework. We
> > could also just add a button to our dashboard which triggers a sub-tree
> ci run. That would help keep narrow
> > the scope of what the email based retesting framework is for. But, both
> email or a dashboard button would
> > both work.
>
> We had discussed this long ago - including agreeing on a format, IIRC.
>
> See the thread starting here:
>   https://mails.dpdk.org/archives/ci/2021-May/001189.html
>
> The idea was to have a line like:
>
> Recheck-request: <test names>
>
> where <test names> was the tests in the check labels.  In fact, what
> started the discussion was a patch for the pw-ci scripts that
> implemented part of it.
>
> I don't see how to make your proposal as easily parsed.
>
> WDYT?  Can you re-read that thread and come up with comments?
>
> > On Tue, Jun 6, 2023 at 1:53 PM Ferruh Yigit <ferruh.yigit at amd.com>
> wrote:
> >
> >  On 6/6/2023 5:56 PM, Patrick Robb wrote:
> >  > Hello all,
> >  >
> >  > I'd like to revive the conversation about a request from the community
> >  > for an email based re-testing framework. The idea is that using one
> >  > standardized format, dpdk developers could email the test-report
> mailing
> >  > list, requesting a rerun on their patch series for "X" set of tests at
> >  > "Y" lab. I think that since patchwork testing labels (ie.
> >  > iol-broadcom-Performance, github-robot: build, loongarch-compilation)
> >  > are already visible on patch pages on patchwork, those labels are the
> >  > most reasonable ones to expect developers to use when requesting a
> >  > re-test. We probably wouldn't want to get any more general than that,
> >  > like, say, rerunning all CI testing for a specific patch series at a
> >  > specific lab, since it would result in a significant amount of
> "wasted"
> >  > testing capacity.
> >  >
> >  > The standard email format those of us at the Community Lab are
> thinking
> >  > of is like below. Developers would request retests by emailing the
> >  > test-report mailing list with email bodies like:
> >  >
> >  > [RETEST UNH-IOL]
> >  > iol-abi-testing
> >  > iol-broadcom-Performance
> >  >
> >  > [RETEST Intel]
> >  > intel-Functional
> >  >
> >  > [RETEST Loongson]
> >  > loongarch-compilation
> >  >
> >  > [RETEST GHA]
> >  > github-robot: build
> >  >
> >  > From there, it would be up to the various labs to poll the test-report
> >  > mailing list archive (or use a similar method) to check for such
> >  > requests, and trigger a CI testing rerun based on the labels provided
> in
> >  > the re-test email. If there is interest from other labs, UNH might
> also
> >  > be able to host the entire set of re-test requests, allowing other
> labs
> >  > to poll a curated list hosted by UNH. One simple approach would be for
> >  > labs to download all emails sent to test-report and parse with regex
> to
> >  > determine the re-test list for their specific lab. But, if anyone has
> >  > any better ideas for aggregating the emails to be parsed, suggestions
> >  > are welcome! If this approach sounds reasonable to everyone, we could
> >  > determine a timeline by which labs would implement the functionality
> >  > needed to trigger re-tests. Or, we can just add re-testing for various
> >  > labs if/when they add this functionality - whatever is better. Happy
> to
> >  > discuss at the CI meeting on Thursday.
> >  >
> >
> >  +1 to re-testing framework.
> >
> >  Also it can be useful to run daily sub-tree testing by request, if
> possible.
>
>

-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mails.dpdk.org/archives/ci/attachments/20230607/eb80682e/attachment.htm>