[dpdk-dev] [PATCH v5 2/2] ci: Introduce travis builds for github repositories

Aaron Conole aconole at redhat.com
Wed Feb 27 15:35:21 CET 2019


Thomas Monjalon <thomas at monjalon.net> writes:

> Hi,
>
> 07/02/2019 23:01, Michael Santana:
>> GitHub is a service used by developers to store repositories.  GitHub
>> provides service integrations that allow 3rd party services to access
>> developer repositories and perform actions.  One of these services is
>> Travis-CI, a simple continuous integration platform.
>> 
>> This is a simple initial implementation of a travis build for the DPDK
>> project.  It doesn't require any changes from individual developers to
>> enable, but will allow those developers who opt-in to GitHub and the
>> travis service to get automatic builds for every push they make.
>> 
>> Additionally, the travis service will send an email to the test-report
>> list informing anyone interested in the automated build (including a
>> result).
>> 
>> Signed-off-by: Aaron Conole <aconole at redhat.com>
>> Signed-off-by: Michael Santana <msantana at redhat.com>
>> Acked-by: Bruce Richardson <bruce.richardson at intel.com>
>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli at arm.com>
>
> I think it is a good idea to provide some recipes for some well known CI.
>
> I have some concerns about the organization of what is generic/specific
> and about the automation of reports.
> Detailed comments below:
>
>> --- /dev/null
>> +++ b/.ci/linux-build.sh
>> +# Just used for the 'classic' configuration system (ie: make)
>
> I am not sure about supporting the legacy system in a new CI.

For now, documentation all says that the legacy system is the supported
system.  I think it's appropriate to continue to support it until such
time as it is eliminated.  Otherwise, when an end user builds we don't
know what to say about support - IE: if they have problems with the
classic system for whatever reason, do we tell them "sorry, we cannot
help"?

The patches.rst mentions that all patches must pass with the Makefile
system, and the contributing/documentation.rst calls it the "standard
DPDK build system."  If you want to change those things to reflect
something different please do, and we can drop all of the stuff related
to it, but until that time we won't.

>> +set_conf() {
>> +    echo "[BUILT WITH $2 SET TO $3]"
>
> In order to improve readability, I suggest to document the parameters
> on the first line. Example:
>
> set_conf() { # <dir> <key> <value>
>
>> +    c="$1/.config"
>> +    shift
>> +
>> +    if grep -q "$1" "$c"; then
>> +        sed -i "s:^$1=.*$:$1=$2:g" $c
>
> You don't need the "g" as there is only one option per line.
>
>> +    else
>> +        echo $1=$2 >> "$c"
>
> The else branch is useless because .config includes all options.

Okay.

>> +    fi
>> +}
>> +
>> +BUILD_ARCH="x86_64-native-linuxapp-"
>
> We could have some native Arm compilation.

Sure.  Is this just commentary?  Do you suggest a change here?  This is
a default, and will be adjusted later by other parameters.

>> +
>> +if [ "${AARCH64}" == "1" ]; then
>> +    # convert the arch specifier
>> +    BUILD_ARCH="arm64-armv8a-linuxapp-"
>> +    AARCH64_TOOL="linaro-arm-tool"
>
> What is this directory? It looks really specific to Travis.

It's specific to the AARCH64 toolchain that was pulled in as part of
linux-prepare.sh - do you think something should change?

>> +    export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin
>> +fi
>> +
>> +
>> +if [ "${NINJABUILD}" == "1" ]; then
>> +    OPTS=""
>> +
>> +    DEF_LIB="static"
>> +    if [ "${SHARED}" == "1" ]; then
>> +        DEF_LIB="shared"
>> +    fi
>> +
>> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
>> +        OPTS="-Denable_kmods=false"
>> +    fi
>
> Isn't it possible to directly provide the meson options in travis.yml
> instead of doing a translation with new option names?

Yes and no.  It would be possible, but we try to support both build
systems and they have different options afaict.  So we need something
common.  Maybe we missed something?

>> +
>> +    if [ "${AARCH64}" == "1" ]; then
>> +        OPTS="${OPTS} --cross-file config/arm/arm64_armv8_linuxapp_gcc"
>> +    fi
>> +
>> +    OPTS="$OPTS --default-library=$DEF_LIB"
>> +    meson build --werror -Dexamples=all ${OPTS}
>> +    ninja -C build
>> +else
>> +    EXTRA_OPTS=""
>> +
>> +    make config T="${BUILD_ARCH}${CC}"
>> +
>> +    set_conf build CONFIG_RTE_KNI_KMOD n
>> +    set_conf build CONFIG_RTE_EAL_IGB_UIO n
>
> Why these options are fixed?

There was a problem with the Travis system when trying to build some of
the kernel modules, but I don't remember the details.  Maybe Michael does.

>> +
>> +    if dpkg --list | grep -q zlib1g ; then
>> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB y
>> +    fi
>
> This is Debian-specific.

Will address this in v6.

>> +
>> +    if dpkg --list | grep -q libpcap-dev ; then
>> +        set_conf build CONFIG_RTE_PORT_PCAP y
>> +    fi
>> +
>> +    if [ "${SHARED}" == "1" ]; then
>> +        set_conf build CONFIG_RTE_BUILD_SHARED_LIB y
>> +    fi
>> +
>> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
>> +        echo Unsupported kernel builds at the moment
>> +    fi
>> +
>> +    if [ "${AARCH64}" == "1" ]; then
>> +        EXTRA_OPTS="CROSS=aarch64-linux-gnu-"
>> +
>> +        # need to turn off these extras
>> +        set_conf build CONFIG_RTE_PORT_PCAP n
>> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB n
>> +
>> +        # convert the CC/CXX variables
>> +        export CC=aarch64-linux-gnu-${CC}
>> +        export CXX=aarch64-linux-gnu-${CXX}
>> +        export AR=aarch64-linux-gnu-ar
>> +        export STRIP=aarch64-linux-gnu-strip
>> +    fi
>> +
>> +    make all ${EXTRA_OPTS}
>> +fi
>> diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh
>> new file mode 100755
>> index 000000000..8dfd5aa49
>> --- /dev/null
>> +++ b/.ci/linux-setup.sh
>> @@ -0,0 +1,31 @@
>> +#!/bin/bash
>> +
>> +python3.5 -m pip install --upgrade meson --user
>
> Which distributions have python3.5?
> It looks very specific.

I agree, could probably just be python3 we need to check and see if we
can just use that.

But, we did need the upgrade.

Travis environment comes up with ubuntu 14.04, which includes python3.4,
and the requisite version of meson needs python3.5 or higher.

>> +
>> +echo "AARCH64 is [ ${AARCH64} ]"
>> +
>> +if [ "${AARCH64}" == "1" ]; then
>> +    # need to build & install libnuma
>
> Why is it needed? linbnuma is optional.
> I think this file can be dropped or renamed to
> something like install-libnuma-for-cross-arm.sh

We needed this because it broke the meson build when cross compiling,
IIRC.  I will investigate it further to be sure.

>> +    # This will only be minimal support for now.
>> +
>> AARCH64_TOOL_URL='https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu.tar.xz'
>> +    AARCH64_TOOL="linaro-arm-tool"
>> +    NUMA_GIT_URL="https://github.com/numactl/numactl.git"
>> +
>> +    wget -O "${AARCH64_TOOL}.tar.xz" "${AARCH64_TOOL_URL}"
>> +    tar -xf "${AARCH64_TOOL}.tar.xz"
>> +    mv gcc-linaro* "${AARCH64_TOOL}"
>> +    export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin
>> +    git clone "${NUMA_GIT_URL}"
>> +    cd numactl
>> +    git checkout v2.0.11
>> +    ./autogen.sh
>> +    autoconf -i
>> +    mkdir numa_bin
>> +    ./configure --host=aarch64-linux-gnu CC=aarch64-linux-gnu-gcc \
>> +                --prefix=$(pwd)/numa_bin
>> +    make install # install numa
>> +    cd ..
>> + cp numactl/numa_bin/include/numa*.h
>> "${AARCH64_TOOL}/aarch64-linux-gnu/libc/usr/include/"
>> +    cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/aarch64-linux-gnu/lib64/"
>> +    cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/lib/"
>> +fi
>> diff --git a/.travis.yml b/.travis.yml
>> new file mode 100644
>> index 000000000..0faa35770
>> --- /dev/null
>> +++ b/.travis.yml
>> @@ -0,0 +1,159 @@
>> +language: c
>> +compiler:
>> +  - gcc
>> +  - clang
>> +
>> +os:
>> +  - linux
>> +
>> +addons:
>> +  apt:
>> +    sources:
>> +      - deadsnakes  #Repo for python 3.5
>> +      - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +    packages:
>> +      - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build]
>> +
>> +before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh
>> +
>> +sudo: false
>> +
>> +env:
>> +  - SHARED=1
>> +  - DISABLE_KERNEL_MODULES=1
>> +  - SHARED=1 DISABLE_KERNEL_MODULES=1
>> +  - NINJABUILD=1
>> +  - NINJABUILD=1 SHARED=1
>> +  - NINJABUILD=1 DISABLE_KERNEL_MODULES=1
>> +  - NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
>> +
>> +matrix:
>> +  include:
>> +  - env: SHARED=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: SHARED=1
>> +    compiler: clang
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: DISABLE_KERNEL_MODULES=1
>> +    compiler: clang
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: clang
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: AARCH64=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), libtool, python3.5, python3-pip]
>> +  - env: AARCH64=1 NINJABUILD=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [linux-headers-$(uname -r), libtool, python3.5, python3-pip, ninja-build]
>> +  - env: NINJABUILD=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: NINJABUILD=1 SHARED=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: NINJABUILD=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +  - env: NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip,
>> ninja-build]
>> +
>> +
>> +script: ./.ci/${TRAVIS_OS_NAME}-build.sh
>> +
>> +notifications:
>> +  email:
>> +    recipients:
>> +      - test-report at dpdk.org
>
> The idea of this mailing list is to receive reports about
> the upstream development. When doing a private development,
> reports should not be sent. How can it be disabled?

There are a few variables related to controlling this that we can
investigate so that we only alert on the main repository, if you want.

We could also just remove it.  Not a problem either way.


More information about the dev mailing list