Email based retest request process: proposal for new pull/re-apply feature

[Aaron Conole]

Patrick Robb <probb at iol.unh.edu> writes:

> Hi all,
>
> I want to poll the CI group and dev community about a proposed feature addition to the CI retest request framework.
> Currently, you can respond to a patchseries or patch email, requesting a retest like so:
>
> Recheck-request: iol-compile-amd64-testing, iol-unit-amd64-testing    
>
> Labs who have added this functionality (UNH and the GitHub Robot) will then trigger retests according to the contexts
> provided, using the ORIGINAL dpdk artifact they produced at the time when the patch was submitted. 
>
> This is useful for requesting a retest on a patch when you believe a failure may have been an infra failure or spurious. It
> is not useful if you believe the tree your patch was applied on was in a bad state when your patch was submitted, and
> you would like for your patch to be re-applied on the current tip of the branch. A few people have suggested we add
> this feature (re-apply to tip of branch and retest). So, we should probably add an option allowing people to indicate they
> want this behavior instead of the "default" retest. 
>
> Ferruh emailed me about this a while ago and proposed the following syntax, which I am okay with:
>
> Recheck-request,attribute=value: ...
>
> So a practical example would look like:
>
> Recheck-request,pull=True: iol-sample-apps-testing, iol-compile-amd64-testing, github-robot
>
> Also, I believe that although we should still require people to include the contexts they're requesting a retest for for
> posterity (so we can refer back to it), I think if someone includes the pull keyword, ALL labs should trigger retests for
> ALL tests. The reason for this is I don't think we should display results side by side on a patchseries which are coming
> from distinct DPDK artifacts. Readers may assume (rightly, in my opinion) that when they're looking at a results table
> for a patchseries, those results are all coming from identical DPDK artifacts, and not from distinct DPDK artifacts
> produced at different times, from different commits.
>
> What do you all think? Thanks.

Why not something like:

Recheck-request: [attribute-list],[test-list]...

For example, then we can do:

Recheck-request: rebase=[identifier],....

where identifier is a branch specifier (or the word 'latest')?

That lets us fixup if the branch picker script guessed a wrong branch.

Just spit-balling on syntax.


That said, I agree - if a rebase has been requested, all tests need to
be rerun.  Maybe we should consider that the test labels should be added
with a run number or something?  Or we could also include that the run
is a rerun.  That way for labs that don't currently support the recheck
request framework, we can easily tell that they weren't re-tried.