<div dir="ltr">Yes, I agree that "specificed" should be corrected to "specified". Thank you for catching this. </div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 10, 2023 at 3:40 PM Aaron Conole <<a href="mailto:aconole@redhat.com">aconole@redhat.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Adam Hassick <<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>> writes: > From: Owen Hilyard <<a href="mailto:ohilyard@iol.unh.edu" target="_blank">ohilyard@iol.unh.edu</a>> > > The Makefile that can be used to build all of the container images using > "make build", and can also be used to push them to a remote repository > (for use in CI). > > Signed-off-by: Owen Hilyard <<a href="mailto:ohilyard@iol.unh.edu" target="_blank">ohilyard@iol.unh.edu</a>> > Signed-off-by: Adam Hassick <<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>> > --- > containers/Makefile | 258 ++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 258 insertions(+) > create mode 100644 containers/Makefile > > diff --git a/containers/Makefile b/containers/Makefile > new file mode 100644 > index 0000000..047a208 > --- /dev/null > +++ b/containers/Makefile > @@ -0,0 +1,258 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright (c) 2023 University of New Hampshire > + > +# Recommended Resources: > +# All: > +# ~100 GB of disk space: These are not minimal containers, they > +# have full distros in them minus the kernel. > +# With ABI images, expect 10 GB per distro per > +# platform. Without, it's closer to 2 GB per distro > +# per platform. > +# > +# No ABI: > +# Serial Build (make -j 1): Should run on a laptop. > +# Parallel build (make -j $(nproc)): 1 GB of free ram per job should be good enough. > +# ABI: > +# Time: If you are building more than four distros, or you have one distro that is built under > +# emulation (x86 host + arm container or the inverse), it may be best to run it overnight. > +# Some versions of qemu + podman will try to compile with a single thread under emulation, > +# which can cause what should be an hour-long compile to turn into a 8+ hour compile. > +# Serial Build (make -j 1): 6 GB of free ram should be fine > +# Parallel build (make -j $(nproc)): 6 GB of free ram per job. You may be compiling ~20 versions of DPDK at the same time, with some under emulation. > +# > + > + > +############ Arguments ############# > +# Set to 'Y' to override distro detection > +DPDK_CI_CONTAINERS_ON_RHEL?=$(shell (test -f /etc/redhat-release && grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') || echo 'N') > + > +# If set to 'Y' and any container is detected as unbuildable, fail the build. > +DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=N > + > +# If set to 'Y', build the ABIs for specificed DPDK versions and embed them If you agree, I can fix this to specified when committing to the repo. > +# in the container > +DPDK_CI_CONTAINERS_BUILD_ABI?=N > + > +# If set to 'Y', the "latest" tag for all images will be omitted. Intended for testing changes to your inventory. > +DPDK_CI_CONTAINERS_NO_LATEST_TAG?=N > + > +# If set to 'Y', will build containers that rely on the Coverity Scan tool. > +DPDK_CI_CONTAINERS_COVERITY?=N > + > +# The path to Coverity Scan binaries. These will be installed inside the container. > +# This field is only required if the Coverity flag is enabled. > +# DPDK_CI_CONTAINERS_COVERITY_PATH?=/opt/dpdklab/coverity > + > +# If set to a non-empty value, overrides the auto-generated date tag with the value. > +# DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE=70-01-01 > + > +# If set to 'Y', build libabigail from source for distros that do not have it. > +# If DPDK_CI_CONTAINERS_BUILD_ABI='Y', then this is always enabled. > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) > +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=Y > +else > +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL?=N > +endif > + > +# If set to an integer value, this will restrict the count of Ninja workers performing the ABI build to the given integer. > +# The argument is benign if DPDK_CI_CONTANERS_BUILD_ABI is 'N'. > +# DPDK_CI_CONTAINERS_NINJA_WORKERS?=16 > + > +# If set to 'Y', only build containers matching the host architecture > +DPDK_CI_CONTAINERS_ONLY_HOST_ARCH?=N > + > +# If set to 'Y', assumes only host arch and disables the push_manifests target. > +DPDK_CI_CONTAINERS_IS_BUILDER?=N > + > +# Used to set the python interpreter > +DPDK_CI_CONTAINERS_PYTHON3_CMD?=python3.8 > + > +# The base program to use to build individual containers. > +DPDK_CI_CONTAINER_BUILDER_PROGRAM?=podman > + > +# The url of the git repository to pull libabigail from > +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=git://<a href="http://sourceware.org/git/libabigail.git" rel="noreferrer" target="_blank">sourceware.org/git/libabigail.git</a> > + > +# The url of the git repository to pull DPDK from > +DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=<a href="https://dpdk.org/git/dpdk" rel="noreferrer" target="_blank">https://dpdk.org/git/dpdk</a> > + > +# The url of the git repository to pull the DPDK CI scripts from. > +DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL?=<a href="https://dpdk.org/git/tools/dpdk-ci" rel="noreferrer" target="_blank">https://dpdk.org/git/tools/dpdk-ci</a> > + > +# The url of the git repository to pull DPDK stable from > +DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL?=<a href="https://dpdk.org/git/dpdk-stable" rel="noreferrer" target="_blank">https://dpdk.org/git/dpdk-stable</a> > + > +# The tag to apply to the built container builder image > +DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder > + > +# Extra arguments to add to the push command, can be used for credentials > +DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?= > + > +# Provide the hostname of the registry to push up to > +DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=localhost > + > +# The path to a directory to be recursively copied to $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY). Not used if unset. > +#DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH?= > + > +DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context > +############ End Arguments ############# > + > +############ Internal Variables ############# > +SCRIPT_ARGS=--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)" > + > +ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y) > + SCRIPT_ARGS +=--rhel > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y) > + SCRIPT_ARGS +=--fail-on-unbuildable > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) > + SCRIPT_ARGS +=--build-abi > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL), Y) > + SCRIPT_ARGS +=--build-libabigail > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH), Y) > + SCRIPT_ARGS +=--host-arch-only > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_NO_LATEST_TAG), Y) > + SCRIPT_ARGS +=--omit-latest > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_IS_BUILDER), Y) > + SCRIPT_ARGS +=--builder-mode > +endif > + > +ifneq ($(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE),) > + SCRIPT_ARGS +=--date $(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE) > +endif > + > +ifneq ($(DPDK_CI_CONTAINERS_NINJA_WORKERS),) > + SCRIPT_ARGS +=--ninja-workers $(DPDK_CI_CONTAINERS_NINJA_WORKERS) > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_COVERITY), Y) > + SCRIPT_ARGS +=--coverity > +endif > + > +SCRIPT_DIRECTORY=$(CURDIR)/template_engine > + > +LIBABIGAIL_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabigail > +DPDK_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk > +DPDK_CI_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-ci > +DPDK_STABLE_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-stable > + > +DOCKERFILE=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile > + > +MAKE_DOCKERFILE_SCRIPT=$(SCRIPT_DIRECTORY)/make_dockerfile.py > +MAKE_DOCKERFILE_COMMAND=$(DPDK_CI_CONTAINERS_PYTHON3_CMD) $(MAKE_DOCKERFILE_SCRIPT) > + > +TEMPLATE_FILE_DIRECTORY=$(SCRIPT_DIRECTORY)/templates > +DOCKER_CONTAINER_TEMPLATE_DIR=$(TEMPLATE_FILE_DIRECTORY)/containers > + > +INVENTORY_FILE=$(SCRIPT_DIRECTORY)/inventory.yaml > + > +EXTRA_SCRIPTS_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scripts > + > +COVERITY_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/coverity > + > +GIT_FETCH_ARGS=--all --tags > + > +CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile > + > +export > +############ End Internal Variables ############# > + > +.PHONY: build push push_images push_manifests make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk dpdk-ci coverity clean_container_files clean > + > +build: make_docker_files_in_container external_files > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) build_all > + > +push_images: make_docker_files_in_container external_files > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_image_all > + > +push_manifests: make_docker_files_in_container > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_manifest_all > + > +push: push_images push_manifests > + > +make_docker_files_in_container: build_builder_container $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run --rm -v $(CURDIR):/container_workspace:z -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context:z \ > + -e DPDK_CI_CONTAINERS_ON_RHEL=$(DPDK_CI_CONTAINERS_ON_RHEL) \ > + -e DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=$(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL) \ > + -e DPDK_CI_CONTAINERS_BUILD_ABI=$(DPDK_CI_CONTAINERS_BUILD_ABI) \ > + -e DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE=$(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE) \ > + -e DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME='$(DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME)' \ > + -e DPDK_CI_CONTAINERS_ONLY_HOST_ARCH='$(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH)' \ > + -e DPDK_CI_CONTAINERS_NO_LATEST_TAG='$(DPDK_CI_CONTAINERS_NO_LATEST_TAG)' \ > + -e DPDK_CI_CONTAINERS_IS_BUILDER='$(DPDK_CI_CONTAINERS_IS_BUILDER)' \ > + -e DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE='$(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE)' \ > + -e DPDK_CI_CONTAINERS_NINJA_WORKERS='$(DPDK_CI_CONTAINERS_NINJA_WORKERS)' \ > + -e DPDK_CI_CONTAINERS_COVERITY='$(DPDK_CI_CONTAINERS_COVERITY)' \ > + $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) make docker_deps > + > +external_files: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) extra_scripts coverity > + > +build_builder_container: > + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f $(CONTAINER_BUILDER_DOCKERFILE) -t $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) $(CURDIR) > + > +docker_deps: $(DOCKERFILE) deps extra_scripts > + chmod 666 $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/*.dockerfile > + > +$(DOCKERFILE): $(INVENTORY_FILE) $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) deps $(MAKE_DOCKERFILE_SCRIPT) $(shell find $(DOCKER_CONTAINER_TEMPLATE_DIR) -type f) > + cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFILE_COMMAND) $(SCRIPT_ARGS) > + > +deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk dpdk-ci libabigail dpdk-stable > + > +extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(EXTRA_SCRIPTS_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) > +ifdef DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH > + cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)/* $(EXTRA_SCRIPTS_DIRECTORY) > +endif > + > +# Clone libabigail source > +libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(LIBABIGAIL_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL) $(LIBABIGAIL_DIRECTORY) > + git -C $(LIBABIGAIL_DIRECTORY) describe --tags `git -C $(LIBABIGAIL_DIRECTORY) rev-list --tags --max-count=1` | xargs -n 1 git -C $(LIBABIGAIL_DIRECTORY) checkout > + > +# Clone DPDK source > +dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY) > + > +# Clone DPDK CI tools source. > +dpdk-ci: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(DPDK_CI_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL) $(DPDK_CI_DIRECTORY) > + > +# Copy the Coverity Scan binaries from a source into the context directory. > +coverity: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(COVERITY_DIRECTORY) > +ifeq ($(DPDK_CI_CONTAINERS_COVERITY),Y) > +ifndef DPDK_CI_CONTAINERS_COVERITY_PATH > + exit 1 # The coverity path must be set if building Coverity containers is enabled. > +else > + cp -pur $(DPDK_CI_CONTAINERS_COVERITY_PATH)/* $(COVERITY_DIRECTORY) > +endif > +else > + # Coverity is disabled for this run. > +endif > + > +dpdk-stable: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(DPDK_STABLE_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL) $(DPDK_STABLE_DIRECTORY) > + > +$(EXTRA_SCRIPTS_DIRECTORY): > + mkdir -p $(EXTRA_SCRIPTS_DIRECTORY) > + > +$(COVERITY_DIRECTORY): > + mkdir -p $(COVERITY_DIRECTORY) > + > +$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY): > + mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + > +clean_container_files: > + rm $(DOCKERFILE) > + > +clean: > + rm -rf $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) </blockquote></div> -- <div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div>Adam Hassick </div></div><div>Senior Developer</div><div>UNH InterOperability Lab</div><div><a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a> </div><div><a href="https://www.iol.unh.edu/" target="_blank">iol.unh.edu</a> </div>+1 (603) 475-8248 </div></div>