[dpdk-dev] [EXT] [PATCH v3 0/7] test: refactor crypto unit test framework

[Akhil Goyal]

Hi Ciara,

I see that case " test_snow3g_auth_cipher_test_case_1 " was unsupported
on marvell PMDs  and was getting skipped, but now after your series
it is causing segmentation fault. I guess there is some check missed for
digest encrypted case.

Can you check?

> The current crypto unit test framework is not granular enough to
> accurately track unit test results. This is caused by one testcase
> in a suite actually running multiple testcases, but only returning
> one result.
> 
> The approach taken in this patchset allows a test suite have a
> list of sub-testsuites, and/or a list of testcases as previously used.
> The unit test suite runner can then recursively iterate and run the
> sub-testsuites, until it reaches a suite with testcases,
> and it then runs each testcase as it had done previously.
> In the case of a testsuite with both testcases and sub-testsuites,
> the testcases are executed first before iterating through the
> sub-testsuites.
> 
> By allowing this further breakdown into sub-testsuites,
> a refactor of the crypto unit tests solves the issue of inaccurate
> reporting, as sub-testsuites can be used in place of the testcases
> that had multiple testcases hidden on a sub level.
> The blockcipher tests previously had these hidden testcases,
> but are now sub-testsuites that are dynamically created and added to a
> parent test suite, allowing for each testcase status to be reported
> directly to the runner.
> The cryptodev test suite is broken down into smaller suites that are
> used as sub-testsuites, which allows for more flexibility choosing which
> sub-testsuites should run for the current device.
> The introduction of sub-testsuites also allows for more precise
> setup/teardown functions, that can check the capabilities required to
> run its testcases.
> 
> For example, when running the cryptodev_aesni_mb_autotest,
> the parent Cryptodev Test Suite is executed.
> Various sub-testsuites are added to the parent test suite, such as
> the static suites of testcases that were once in the cryptodev_testsuite,
> and blockcipher suites.
> The unit test runner can then run the Cryptodev parent test suite,
> which in turn will run the sub-testsuites.
> 
> The user is now required to create vdevs via EAL commandline args,
> this is no longer done within the test app for crypto autotests.
> 
> Documentation will need to be added at a later stage,
> adding to the test document that isn't yet merged. [1]
> 
> [1] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__patchwork.dpdk.org_project_dpdk_patch_20210309155757.615536-
> 2D1-2Daconole-
> 40redhat.com_&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si2wl_PR
> wpZ9TWey3eu68gBzn7DkPwuqhd6WNyo&m=KnksoVYw74RhGrlmzJAVAS_O
> uxTVx-7GPFYdyP-qn14&s=uvfBln2-zGHif5yvsK0_QuZhdo4e-
> 3MpVXIevmc5OBE&e=
> 
> 
> v3:
>   - Added support for a testsuite having both a list of testcases,
>     and a list of sub-testsuites.
>   - Replaced PMD based parent testsuites with a cryptodev testsuite
>     used by all autotests, with the exception of scheduler autotest.
>   - Setup functions were added for all sub-testsuites, within which
>     required capability support is checked.
>   - The setup functions no longer create vdevs if needed,
>     this must be done by the user when running the test.
>   - Patch added to standardise return values for skipped testcases.
> v2:
>   - Added macro in place of testcase/testsuite loops.
>   - Added more detail in the summary output.
>   - Moved testcase counts to the testsuite structure.
>   - Flattened testsuite structure to remove union.
>   - Added patch for fix of blockcipher test return value.
>   - Squashed release note into last patch.
> 
> 
> Ciara Power (7):
>   app/test: refactor of unit test suite runner
>   test: introduce parent testsuite format
>   test/crypto: refactor to use sub-testsuites
>   test/crypto: replace unsupported with skipped
>   test/crypto: move testsuite params to header file
>   test/crypto: fix return value on test skipped
>   test/crypto: dynamically build blockcipher suite
> 
>  app/test/test.c                        |  176 +-
>  app/test/test.h                        |   23 +-
>  app/test/test_cryptodev.c              | 2324 ++++++++++++++----------
>  app/test/test_cryptodev.h              |   32 +
>  app/test/test_cryptodev_asym.c         |  111 +-
>  app/test/test_cryptodev_blockcipher.c  |  423 ++++-
>  app/test/test_cryptodev_blockcipher.h  |   12 +-
>  app/test/test_ipsec.c                  |   32 +-
>  doc/guides/rel_notes/release_21_05.rst |    5 +
>  9 files changed, 2000 insertions(+), 1138 deletions(-)
> 
> --
> 2.25.1