[PATCH v5 04/10] app/test: build using per-file dependency matrix

[Bruce Richardson]

On Thu, Aug 17, 2023 at 09:27:50AM +0200, David Marchand wrote:
> On Wed, Aug 16, 2023 at 8:30 PM Patrick Robb <probb at iol.unh.edu> wrote:
> >> Patching sources from the test tool is a poor solution.
> >> In general, developers won't be aware of source patching and will
> >> waste time trying to understand why they can't reproduce what the CI
> >> reports (it happened to me with DTS on the interrupt stuff with vhost,
> >> at least).
> >>
> >> For this specific case of skipping a test, if nobody can fix the
> >> issue, I prefer if the CI can skip some "known broken in my lab" tests
> >> via some meson configuration.
> >> And, such configuration should be easy to catch in the test report.
> >>
> > I strongly agree on all points, which is why I said it was probably a good thing anyhow for us to lose this ability. In the case of the disabled fast-test for arm, that was a new discovery coming from adding new environments, not a regression introduced by a patch, and I don't think it made sense then to block the introduction of the entire unit test coverage for arm while they looked into this issue. If it's possible to introduce meson configure functionality to disable specific tests, that does give us more flexibility. And it's obviously a better process than us doing it at the CI end.
> >
> > We don't currently patch source in any other way in our CI testing.
> 
> It is possible to list tests from meson.
> Why not simply list the existing tests and filter the non working one?
> 
> $ meson test -C <build_dir> --suite fast-tests --list
> $ meson test -C <build_dir> <explicit list> --test-args ...
> 
> That requires no change in DPDK.
> 
The small issue with that is that the broken test is hidden from reports.
While it unblocks things here without having to change DPDK, I wonder if
changing DPDK to support this scenario might be worthwhile. For example, one
idea I'm now thinking of is to have the test binary itself look in the
environment for a DPDK_TEST_SKIPPED variable, and - when processing test
names passed on command-line or environment - report any on that list as
SKIPPED. This at least flags in the reports that a test is not running, and
the test log gives a reason. 

Worth doing? Or is adding a new environment with broken tests rare enough
we don't need this?

Of course, this also seems like a feature request for meson. When running
all tests, meson allows you to skip a test suite, but doesn't allow
skipping a test within a suite.

/Bruce