Email Based Re-Testing Framework

[Ferruh Yigit]

On 6/6/2023 8:27 PM, Patrick Robb wrote:
>     Also it can be useful to run daily sub-tree testing by request, if
>     possible.
> 
> 
> That wouldn't be too difficult. I'll make a ticket for this. Although,
> for testing on the daily sub-trees, since that's UNH-IOL specific, that
> wouldn't necessarily have to be done via an email based testing request
> framework. We could also just add a button to our dashboard which
> triggers a sub-tree ci run. That would help keep narrow the scope of
> what the email based retesting framework is for. But, both email or a
> dashboard button would both work. 
> 

Thanks, agree that for sub-trees a button on dashboard is sufficient.


> On Tue, Jun 6, 2023 at 1:53 PM Ferruh Yigit <ferruh.yigit at amd.com
> <mailto:ferruh.yigit at amd.com>> wrote:
> 
>     On 6/6/2023 5:56 PM, Patrick Robb wrote:
>     > Hello all,
>     >
>     > I'd like to revive the conversation about a request from the community
>     > for an email based re-testing framework. The idea is that using one
>     > standardized format, dpdk developers could email the test-report
>     mailing
>     > list, requesting a rerun on their patch series for "X" set of tests at
>     > "Y" lab. I think that since patchwork testing labels (ie.
>     > iol-broadcom-Performance, github-robot: build, loongarch-compilation)
>     > are already visible on patch pages on patchwork, those labels are the
>     > most reasonable ones to expect developers to use when requesting a
>     > re-test. We probably wouldn't want to get any more general than that,
>     > like, say, rerunning all CI testing for a specific patch series at a
>     > specific lab, since it would result in a significant amount of
>     "wasted"
>     > testing capacity.
>     >
>     > The standard email format those of us at the Community Lab are
>     thinking
>     > of is like below. Developers would request retests by emailing the
>     > test-report mailing list with email bodies like:
>     >
>     > [RETEST UNH-IOL]
>     > iol-abi-testing
>     > iol-broadcom-Performance
>     >
>     > [RETEST Intel]
>     > intel-Functional
>     >
>     > [RETEST Loongson]
>     > loongarch-compilation
>     >
>     > [RETEST GHA]
>     > github-robot: build
>     >
>     > From there, it would be up to the various labs to poll the test-report
>     > mailing list archive (or use a similar method) to check for such
>     > requests, and trigger a CI testing rerun based on the labels
>     provided in
>     > the re-test email. If there is interest from other labs, UNH might
>     also
>     > be able to host the entire set of re-test requests, allowing other
>     labs
>     > to poll a curated list hosted by UNH. One simple approach would be for
>     > labs to download all emails sent to test-report and parse with
>     regex to
>     > determine the re-test list for their specific lab. But, if anyone has
>     > any better ideas for aggregating the emails to be parsed, suggestions
>     > are welcome! If this approach sounds reasonable to everyone, we could
>     > determine a timeline by which labs would implement the functionality
>     > needed to trigger re-tests. Or, we can just add re-testing for various
>     > labs if/when they add this functionality - whatever is better.
>     Happy to
>     > discuss at the CI meeting on Thursday.
>     >
> 
>     +1 to re-testing framework.
> 
> 
>     Also it can be useful to run daily sub-tree testing by request, if
>     possible.
> 
> 
> 
> -- 
> 
> Patrick Robb
> 
> Technical Service Manager
> 
> UNH InterOperability Laboratory
> 
> 21 Madbury Rd, Suite 100, Durham, NH 03824
> 
> www.iol.unh.edu <http://www.iol.unh.edu/>
> 
>