[dpdk-dev] [PATCH v2 1/2] ci: hook to GitHub Actions

Aaron Conole aconole at redhat.com
Mon Dec 14 15:12:31 CET 2020


David Marchand <david.marchand at redhat.com> writes:

> With the recent changes in terms of free access to the Travis CI, let's
> offer an alternative with GitHub Actions.
> Running jobs on ARM is not supported unless using external runners, so
> this commit only adds builds for x86_64 and cross compiling for i386 and
> aarch64.
>
> Differences with the Travis CI integration:
> - Error logs are not dumped to the console when something goes wrong.
>   Instead, they are gathered in a "catch-all" step and attached as
>   artifacts.
> - A cache entry is stored once and for all, but if no cache is found you
>   can inherit from the default branch cache. The cache is 5GB large, for
>   the whole git repository.
> - The maximum retention of logs and artifacts is 3 months.
> - /home/runner is world writable, so a workaround has been added for
>   starting dpdk processes.
> - Ilya, working on OVS GHA support, noticed that jobs can run with
>   processors that don't have the same capabilities. For DPDK, this
>   impacts the ccache content since everything was built with
>   -march=native so far, and we will end up with binaries that can't run
>   in a later build. The problem has not been seen in Travis CI (?) but
>   it is safer to use a fixed "-Dmachine=default" in any case.

That's because the build machine and test machine are the same, but I
think GHA uses a different model, and will spawn a new environment for
the steps.  I'm not 100% sure, because it's all supposed to be a
black-box.

> - Scheduling jobs is part of the configuration and takes the form of a
>   crontab. A build is scheduled every Monday at 0:00 (UTC) to provide a
>   default ccache for the week (useful for the ovsrobot).
>
> Signed-off-by: David Marchand <david.marchand at redhat.com>
> ---

Acked-by: Aaron Conole <aconole at redhat.com>

> Changelog since v1:
> - changed shell variables value in CI scripts and Travis configuration
>   (s/=[^\$]*1/=\1true), this makes it easier for GHA,
> - forced compilation as 'default' to avoid random unit tests issues in
>   GHA,
> - scheduled a run per week on Monday at 0:00 UTC,
> - updated the ccache key:
>   - no need to depend on the default-library parameter since this
>     parameter only impacts the linking of dpdk binaries,
>   - the week when the cache is generated is added so that jobs in
>     other branches can benefit from a recent cache (mimicking what we had
>     for the robot in Travis),
> - realigned documentation generation with what is done in Travis:
>   generating the doc in all jobs was a waste of resources,
>
> ---
>  .ci/linux-build.sh          |  17 +++---
>  .github/workflows/build.yml | 100 ++++++++++++++++++++++++++++++++++++
>  .travis.yml                 |  24 ++++-----
>  MAINTAINERS                 |   1 +
>  4 files changed, 123 insertions(+), 19 deletions(-)
>  create mode 100644 .github/workflows/build.yml
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index d079801d78..ee8d07f865 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -12,7 +12,9 @@ on_error() {
>          fi
>      done
>  }
> -trap on_error EXIT
> +# We capture the error logs as artifacts in Github Actions, no need to dump
> +# them via a EXIT handler.
> +[ -n "$GITHUB_WORKFLOW" ] || trap on_error EXIT
>  
>  install_libabigail() {
>      version=$1
> @@ -28,16 +30,16 @@ install_libabigail() {
>      rm ${version}.tar.gz
>  }
>  
> -if [ "$AARCH64" = "1" ]; then
> +if [ "$AARCH64" = "true" ]; then
>      # convert the arch specifier
>      OPTS="$OPTS --cross-file config/arm/arm64_armv8_linux_gcc"
>  fi
>  
> -if [ "$BUILD_DOCS" = "1" ]; then
> +if [ "$BUILD_DOCS" = "true" ]; then
>      OPTS="$OPTS -Denable_docs=true"
>  fi
>  
> -if [ "$BUILD_32BIT" = "1" ]; then
> +if [ "$BUILD_32BIT" = "true" ]; then
>      OPTS="$OPTS -Dc_args=-m32 -Dc_link_args=-m32"
>      export PKG_CONFIG_LIBDIR="/usr/lib32/pkgconfig"
>  fi
> @@ -48,16 +50,17 @@ else
>      OPTS="$OPTS -Dexamples=all"
>  fi
>  
> +OPTS="$OPTS -Dmachine=default"
>  OPTS="$OPTS --default-library=$DEF_LIB"
>  OPTS="$OPTS --buildtype=debugoptimized"
>  meson build --werror $OPTS
>  ninja -C build
>  
> -if [ "$AARCH64" != "1" ]; then
> +if [ "$AARCH64" != "true" ]; then
>      devtools/test-null.sh
>  fi
>  
> -if [ "$ABI_CHECKS" = "1" ]; then
> +if [ "$ABI_CHECKS" = "true" ]; then
>      LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
>  
>      if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> @@ -95,6 +98,6 @@ if [ "$ABI_CHECKS" = "1" ]; then
>      devtools/check-abi.sh reference install ${ABI_CHECKS_WARN_ONLY:-}
>  fi
>  
> -if [ "$RUN_TESTS" = "1" ]; then
> +if [ "$RUN_TESTS" = "true" ]; then
>      sudo meson test -C build --suite fast-tests -t 3
>  fi
> diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
> new file mode 100644
> index 0000000000..bef6e52372
> --- /dev/null
> +++ b/.github/workflows/build.yml
> @@ -0,0 +1,100 @@
> +name: build
> +
> +on:
> +  push:
> +  schedule:
> +    - cron: '0 0 * * 1'
> +
> +defaults:
> +  run:
> +    shell: bash --noprofile --norc -exo pipefail {0}
> +
> +jobs:
> +  build:
> +    name: ${{ join(matrix.config.*, '-') }}
> +    runs-on: ${{ matrix.config.os }}
> +    env:
> +      AARCH64: ${{ matrix.config.cross == 'aarch64' }}
> +      BUILD_32BIT: ${{ matrix.config.cross == 'i386' }}
> +      BUILD_DOCS: ${{ contains(matrix.config.checks, 'doc') }}
> +      CC: ccache ${{ matrix.config.compiler }}
> +      DEF_LIB: ${{ matrix.config.library }}
> +      RUN_TESTS: ${{ contains(matrix.config.checks, 'tests') }}
> +
> +    strategy:
> +      fail-fast: false
> +      matrix:
> +        config:
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: shared
> +            checks: doc+tests
> +          - os: ubuntu-18.04
> +            compiler: clang
> +            library: static
> +          - os: ubuntu-18.04
> +            compiler: clang
> +            library: shared
> +            checks: doc+tests
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +            cross: i386
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +            cross: aarch64
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: shared
> +            cross: aarch64
> +
> +    steps:
> +    - name: Checkout sources
> +      uses: actions/checkout at v2
> +    - name: Generate cache keys
> +      id: get_ref_keys
> +      run: |
> +        echo -n '::set-output name=ccache::'
> +        echo 'ccache-${{ matrix.config.os }}-${{ matrix.config.compiler }}-${{ matrix.config.cross }}-'$(date -u +%Y-w%W)
> +    - name: Retrieve ccache cache
> +      uses: actions/cache at v2
> +      with:
> +        path: ~/.ccache
> +        key: ${{ steps.get_ref_keys.outputs.ccache }}-${{ github.ref }}
> +        restore-keys: |
> +          ${{ steps.get_ref_keys.outputs.ccache }}-refs/heads/main
> +    - name: Install packages
> +      run: sudo apt install -y ccache libnuma-dev python3-setuptools
> +        python3-wheel python3-pip ninja-build libbsd-dev libpcap-dev
> +        libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
> +    - name: Install i386 cross compiling packages
> +      if: env.BUILD_32BIT == 'true'
> +      run: sudo apt install -y gcc-multilib
> +    - name: Install aarch64 cross compiling packages
> +      if: env.AARCH64 == 'true'
> +      run: sudo apt install -y gcc-aarch64-linux-gnu libc6-dev-arm64-cross
> +        pkg-config-aarch64-linux-gnu
> +    - name: Install doc generation packages
> +      if: env.BUILD_DOCS == 'true'
> +      run: sudo apt install -y doxygen graphviz python3-sphinx
> +        python3-sphinx-rtd-theme
> +    - name: Run setup
> +      run: |
> +        .ci/linux-setup.sh
> +        # Workaround on $HOME permissions as EAL checks them for plugin loading
> +        chmod o-w $HOME
> +    - name: Build and test
> +      run: .ci/linux-build.sh
> +    - name: Upload logs on failure
> +      if: failure()
> +      uses: actions/upload-artifact at v2
> +      with:
> +        name: meson-logs-${{ join(matrix.config.*, '-') }}
> +        path: |
> +          build/meson-logs/testlog.txt
> +          build/.ninja_log
> +          build/meson-logs/meson-log.txt
> diff --git a/.travis.yml b/.travis.yml
> index 5e12db23b5..d655e286c3 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -34,10 +34,10 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="shared" BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=true
>      arch: amd64
>      compiler: gcc
>      addons:
> @@ -49,10 +49,10 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="shared" BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=true
>      arch: amd64
>      compiler: clang
>      addons:
> @@ -61,7 +61,7 @@ jobs:
>            - *required_packages
>            - *doc_packages
>    # x86_64 cross-compiling 32-bits jobs
> -  - env: DEF_LIB="static" BUILD_32BIT=1
> +  - env: DEF_LIB="static" BUILD_32BIT=true
>      arch: amd64
>      compiler: gcc
>      addons:
> @@ -69,14 +69,14 @@ jobs:
>          packages:
>            - *build_32b_packages
>    # x86_64 cross-compiling aarch64 jobs
> -  - env: DEF_LIB="static" AARCH64=1
> +  - env: DEF_LIB="static" AARCH64=true
>      arch: amd64
>      compiler: gcc
>      addons:
>        apt:
>          packages:
>            - *aarch64_packages
> -  - env: DEF_LIB="shared" AARCH64=1
> +  - env: DEF_LIB="shared" AARCH64=true
>      arch: amd64
>      compiler: gcc
>      addons:
> @@ -87,16 +87,16 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: arm64
>      compiler: gcc
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: arm64
>      compiler: gcc
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      dist: focal
>      arch: arm64-graviton2
>      virt: vm
>      group: edge
>      compiler: gcc
> -  - env: DEF_LIB="shared" BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=true
>      arch: arm64
>      compiler: gcc
>      addons:
> @@ -108,10 +108,10 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: arm64
>      compiler: clang
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: arm64
>      compiler: clang
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      dist: focal
>      arch: arm64-graviton2
>      virt: vm
> diff --git a/MAINTAINERS b/MAINTAINERS
> index eafe9f8c46..f45c8c1b13 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -109,6 +109,7 @@ Public CI
>  M: Aaron Conole <aconole at redhat.com>
>  M: Michael Santana <maicolgabriel at hotmail.com>
>  F: .travis.yml
> +F: .github/workflows/build.yml
>  F: .ci/
>  
>  ABI Policy & Versioning



More information about the dev mailing list