Email Based Re-Testing Framework

[Ali Alnubani]

On 6/6/2023 7:57 PM, Patrick Robb wrote:
> Hello all,
> 
> I'd like to revive the conversation about a request from the community
> for an email based re-testing framework. The idea is that using one
> standardized format, dpdk developers could email the test-report mailing
> list, requesting a rerun on their patch series for "X" set of tests at
> "Y" lab. I think that since patchwork testing labels (ie.
> iol-broadcom-Performance, github-robot: build, loongarch-compilation)
> are already visible on patch pages on patchwork, those labels are the
> most reasonable ones to expect developers to use when requesting a
> re-test. We probably wouldn't want to get any more general than that,
> like, say, rerunning all CI testing for a specific patch series at a
> specific lab, since it would result in a significant amount of "wasted"
> testing capacity.
> 
> The standard email format those of us at the Community Lab are thinking
> of is like below. Developers would request retests by emailing the
> test-report mailing list with email bodies like:
> 
> [RETEST UNH-IOL]
> iol-abi-testing
> iol-broadcom-Performance
> 
> [RETEST Intel]
> intel-Functional
> 
> [RETEST Loongson]
> loongarch-compilation
> 
> [RETEST GHA]
> github-robot: build
> 
> From there, it would be up to the various labs to poll the test-report
> mailing list archive (or use a similar method) to check for such
> requests, and trigger a CI testing rerun based on the labels provided in
> the re-test email. If there is interest from other labs, UNH might also
> be able to host the entire set of re-test requests, allowing other labs
> to poll a curated list hosted by UNH. One simple approach would be for
> labs to download all emails sent to test-report and parse with regex to
> determine the re-test list for their specific lab. But, if anyone has
> any better ideas for aggregating the emails to be parsed, suggestions
> are welcome! If this approach sounds reasonable to everyone, we could
> determine a timeline by which labs would implement the functionality
> needed to trigger re-tests. Or, we can just add re-testing for various
> labs if/when they add this functionality - whatever is better. Happy to
> discuss at the CI meeting on Thursday.
>

Hello,

For context, and as discussed in the last community CI meeting, going through every new patch to look for new comments that trigger retests might take too long and potentially slow down the server.

I will upgrade Patchwork to v3.1 right after the v23.07 release.
The new version adds two new events to the /events API: cover-comment-created and patch-comment-created.[1]

An example event schema:

"""
{
        [..]
        "category": "patch-comment-created",
        "project": {
            [..]
        },
        "date": "string",
        "actor": {
            [..]
        },
        "payload": {
            "patch": {
                [..]
            },
            "comment": {
                [..]
                "url": "https://patches.dpdk.org/api/patches/X/comments/Y/",
                [..]
            }
        }
    }
"""

The comments body/contents can be extracted from the "content" property after fetching the comment's api url. Example schema:

"""
{
    [..]
    "subject": "string",
    "submitter": {
        [..]
    },
    "content": "string",
    "headers": {
        [..]
    },
    "addressed": null
}
"""

[1] https://patchwork.readthedocs.io/en/latest/releases/hessian/#relnotes-v3-1-0-stable-3-1-new-features
Also see:
https://patchwork.readthedocs.io/en/latest/api/rest/schemas/v1.2/#get--api-1.2-events-
https://patchwork.readthedocs.io/en/latest/api/rest/schemas/v1.2/#get--api-1.2-patches-id-comments-

Let me know if you have any questions.

Regards,
Ali