patch 'doc: fix AF_XDP device plugin howto' has been queued to stable release 23.11.2
Xueming Li
xuemingl at nvidia.com
Fri Jul 12 12:44:02 CEST 2024
Hi,
FYI, your patch has been queued to stable release 23.11.2
Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet.
It will be pushed if I get no objections before 07/14/24. So please
shout if anyone has objections.
Also note that after the patch there's a diff of the upstream commit vs the
patch applied to the branch. This will indicate if there was any rebasing
needed to apply to the stable branch. If there were code changes for rebasing
(ie: not only metadata diffs), please double check that the rebase was
correctly done.
Queued patches are on a temporary branch at:
https://git.dpdk.org/dpdk-stable/log/?h=23.11-staging
This queued commit can be viewed at:
https://git.dpdk.org/dpdk-stable/commit/?h=23.11-staging&id=190a4d4844aeca2c94e4b0f49065057928bf202e
Thanks.
Xueming Li <xuemingl at nvidia.com>
---
>From 190a4d4844aeca2c94e4b0f49065057928bf202e Mon Sep 17 00:00:00 2001
From: Maryam Tahhan <mtahhan at redhat.com>
Date: Mon, 8 Apr 2024 09:09:20 -0400
Subject: [PATCH] doc: fix AF_XDP device plugin howto
Cc: Xueming Li <xuemingl at nvidia.com>
[ upstream commit 696982d5efd1ac40576c076b4358a144181d4a0e ]
Fixup the references to the AF_XDP Device Plugin in
the documentation (was referred to as CNI previously)
and document the single netdev limitation for deploying
an AF_XDP based DPDK pod. Also renames af_xdp_cni.rst to
af_xdp_dp.rst
Fixes: 7fc6ae50369d ("net/af_xdp: support CNI Integration")
Signed-off-by: Maryam Tahhan <mtahhan at redhat.com>
Acked-by: Ciara Loftus <ciara.loftus at intel.com>
---
.mailmap | 2 +-
doc/guides/howto/af_xdp_cni.rst | 253 --------------------------
doc/guides/howto/af_xdp_dp.rst | 306 ++++++++++++++++++++++++++++++++
doc/guides/howto/index.rst | 2 +-
doc/guides/nics/af_xdp.rst | 4 +-
5 files changed, 310 insertions(+), 257 deletions(-)
delete mode 100644 doc/guides/howto/af_xdp_cni.rst
create mode 100644 doc/guides/howto/af_xdp_dp.rst
diff --git a/.mailmap b/.mailmap
index f3b429b747..c6bf385a59 100644
--- a/.mailmap
+++ b/.mailmap
@@ -899,7 +899,7 @@ Martin Klozik <martinx.klozik at intel.com>
Martin Spinler <spinler at cesnet.cz>
Martin Weiser <martin.weiser at allegro-packets.com>
Martyna Szapar-Mudlaw <martyna.szapar-mudlaw at intel.com> <martyna.szapar at intel.com>
-Maryam Tahhan <maryam.tahhan at intel.com>
+Maryam Tahhan <mtahhan at redhat.com> <maryam.tahhan at intel.com>
Masoud Hasanifard <masoudhasanifard at gmail.com>
Masoumeh Farhadi Nia <masoumeh.farhadinia at gmail.com>
Matan Azrad <matan at nvidia.com> <matan at mellanox.com>
diff --git a/doc/guides/howto/af_xdp_cni.rst b/doc/guides/howto/af_xdp_cni.rst
deleted file mode 100644
index a1a6d5b99c..0000000000
--- a/doc/guides/howto/af_xdp_cni.rst
+++ /dev/null
@@ -1,253 +0,0 @@
-.. SPDX-License-Identifier: BSD-3-Clause
- Copyright(c) 2023 Intel Corporation.
-
-Using a CNI with the AF_XDP driver
-==================================
-
-Introduction
-------------
-
-CNI, the Container Network Interface, is a technology for configuring
-container network interfaces
-and which can be used to setup Kubernetes networking.
-AF_XDP is a Linux socket Address Family that enables an XDP program
-to redirect packets to a memory buffer in userspace.
-
-This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within
-a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies.
-
-.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
-
-
-Background
-----------
-
-The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program
-onto the kernel netdev to be used by the PMD.
-This operation requires root or escalated Linux privileges
-and thus prevents the PMD from working in an unprivileged container.
-The AF_XDP CNI plugin handles this situation
-by providing a device plugin that performs the program loading.
-
-At a technical level the CNI opens a Unix Domain Socket and listens for a client
-to make requests over that socket.
-A DPDK application acting as a client connects and initiates a configuration "handshake".
-The client then receives a file descriptor which points to the XSKMAP
-associated with the loaded eBPF program.
-The XSKMAP is a BPF map of AF_XDP sockets (XSK).
-The client can then proceed with creating an AF_XDP socket
-and inserting that socket into the XSKMAP pointed to by the descriptor.
-
-The EAL vdev argument ``use_cni`` is used to indicate that the user wishes
-to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor
-from the CNI.
-When this flag is set,
-the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag
-should be used when creating the socket
-to instruct libbpf not to load the default libbpf program on the netdev.
-Instead the loading is handled by the CNI.
-
-.. note::
-
- The Unix Domain Socket file path appear in the end user is "/tmp/afxdp.sock".
-
-
-Prerequisites
--------------
-
-Docker and container prerequisites:
-
-* Set up the device plugin
- as described in the instructions for `AF_XDP Plugin for Kubernetes`_.
-
-* The Docker image should contain the libbpf and libxdp libraries,
- which are dependencies for AF_XDP,
- and should include support for the ``ethtool`` command.
-
-* The Pod should have enabled the capabilities ``CAP_NET_RAW`` and ``CAP_BPF``
- for AF_XDP along with support for hugepages.
-
-* Increase locked memory limit so containers have enough memory for packet buffers.
- For example:
-
- .. code-block:: console
-
- cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf
- [Service]
- LimitMEMLOCK=infinity
- EOF
-
-* dpdk-testpmd application should have AF_XDP feature enabled.
-
- For further information see the docs for the: :doc:`../../nics/af_xdp`.
-
-
-Example
--------
-
-Howto run dpdk-testpmd with CNI plugin:
-
-* Clone the CNI plugin
-
- .. code-block:: console
-
- # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git
-
-* Build the CNI plugin
-
- .. code-block:: console
-
- # cd afxdp-plugins-for-kubernetes/
- # make build
-
- .. note::
-
- CNI plugin has a dependence on the config.json.
-
- Sample Config.json
-
- .. code-block:: json
-
- {
- "logLevel":"debug",
- "logFile":"afxdp-dp-e2e.log",
- "pools":[
- {
- "name":"e2e",
- "mode":"primary",
- "timeout":30,
- "ethtoolCmds" : ["-L -device- combined 1"],
- "devices":[
- {
- "name":"ens785f0"
- }
- ]
- }
- ]
- }
-
- For further reference please use the `config.json`_
-
- .. _config.json: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/config.json
-
-* Create the Network Attachment definition
-
- .. code-block:: console
-
- # kubectl create -f nad.yaml
-
- Sample nad.yml
-
- .. code-block:: yaml
-
- apiVersion: "k8s.cni.cncf.io/v1"
- kind: NetworkAttachmentDefinition
- metadata:
- name: afxdp-e2e-test
- annotations:
- k8s.v1.cni.cncf.io/resourceName: afxdp/e2e
- spec:
- config: '{
- "cniVersion": "0.3.0",
- "type": "afxdp",
- "mode": "cdq",
- "logFile": "afxdp-cni-e2e.log",
- "logLevel": "debug",
- "ipam": {
- "type": "host-local",
- "subnet": "192.168.1.0/24",
- "rangeStart": "192.168.1.200",
- "rangeEnd": "192.168.1.216",
- "routes": [
- { "dst": "0.0.0.0/0" }
- ],
- "gateway": "192.168.1.1"
- }
- }'
-
- For further reference please use the `nad.yaml`_
-
- .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/nad.yaml
-
-* Build the Docker image
-
- .. code-block:: console
-
- # docker build -t afxdp-e2e-test -f Dockerfile .
-
- Sample Dockerfile:
-
- .. code-block:: console
-
- FROM ubuntu:20.04
- RUN apt-get update -y
- RUN apt install build-essential libelf-dev -y
- RUN apt-get install iproute2 acl -y
- RUN apt install python3-pyelftools ethtool -y
- RUN apt install libnuma-dev libjansson-dev libpcap-dev net-tools -y
- RUN apt-get install clang llvm -y
- COPY ./libbpf<version>.tar.gz /tmp
- RUN cd /tmp && tar -xvmf libbpf<version>.tar.gz && cd libbpf/src && make install
- COPY ./libxdp<version>.tar.gz /tmp
- RUN cd /tmp && tar -xvmf libxdp<version>.tar.gz && cd libxdp && make install
-
- .. note::
-
- All the files that need to COPY-ed should be in the same directory as the Dockerfile
-
-* Run the Pod
-
- .. code-block:: console
-
- # kubectl create -f pod.yaml
-
- Sample pod.yaml:
-
- .. code-block:: yaml
-
- apiVersion: v1
- kind: Pod
- metadata:
- name: afxdp-e2e-test
- annotations:
- k8s.v1.cni.cncf.io/networks: afxdp-e2e-test
- spec:
- containers:
- - name: afxdp
- image: afxdp-e2e-test:latest
- imagePullPolicy: Never
- env:
- - name: LD_LIBRARY_PATH
- value: /usr/lib64/:/usr/local/lib/
- command: ["tail", "-f", "/dev/null"]
- securityContext:
- capabilities:
- add:
- - CAP_NET_RAW
- - CAP_BPF
- resources:
- requests:
- hugepages-2Mi: 2Gi
- memory: 2Gi
- afxdp/e2e: '1'
- limits:
- hugepages-2Mi: 2Gi
- memory: 2Gi
- afxdp/e2e: '1'
-
- For further reference please use the `pod.yaml`_
-
- .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/pod-1c1d.yaml
-
-* Run DPDK with a command like the following:
-
- .. code-block:: console
-
- kubectl exec -i <Pod name> --container <containers name> -- \
- /<Path>/dpdk-testpmd -l 0,1 --no-pci \
- --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \
- -- --no-mlockall --in-memory
-
-For further reference please use the `e2e`_ test case in `AF_XDP Plugin for Kubernetes`_
-
- .. _e2e: https://github.com/intel/afxdp-plugins-for-kubernetes/tree/v0.0.2/test/e2e
diff --git a/doc/guides/howto/af_xdp_dp.rst b/doc/guides/howto/af_xdp_dp.rst
new file mode 100644
index 0000000000..2f51b37f20
--- /dev/null
+++ b/doc/guides/howto/af_xdp_dp.rst
@@ -0,0 +1,306 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2023 Intel Corporation.
+
+Using the AF_XDP driver in Kubernetes
+=====================================
+
+Introduction
+------------
+
+Two infrastructure components are needed in order to provision a pod
+that is using the AF_XDP PMD in Kubernetes:
+
+1. AF_XDP Device Plugin (DP).
+2. AF_XDP Container Network Interface (CNI) binary.
+
+Both of these components are available through
+the `AF_XDP Device Plugin for Kubernetes`_ repository.
+
+The AF_XDP DP provisions and advertises networking interfaces to Kubernetes,
+while the CNI configures and plumbs network interfaces for the Pod.
+
+This document explains how to use the `AF_XDP Device Plugin for Kubernetes`_
+with a DPDK application using the :doc:`../nics/af_xdp`.
+
+.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
+
+
+Background
+----------
+
+The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program
+onto the kernel netdev to be used by the PMD.
+This operation requires root or escalated Linux privileges
+and thus prevents the PMD from working in an unprivileged container.
+The AF_XDP Device Plugin handles this situation
+by managing the eBPF program(s) on behalf of the Pod, outside of the pod context.
+
+At a technical level the AF_XDP Device Plugin opens a Unix Domain Socket (UDS)
+and listens for a client to make requests over that socket.
+A DPDK application acting as a client connects and initiates a configuration "handshake".
+After some validation on the Device Plugin side,
+the client receives a file descriptor which points to the XSKMAP
+associated with the loaded eBPF program.
+The XSKMAP is an eBPF map of AF_XDP sockets (XSK).
+The client can then proceed with creating an AF_XDP socket
+and inserting that socket into the XSKMAP pointed to by the descriptor.
+
+The EAL vdev argument ``use_cni`` is used to indicate that the user wishes
+to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor
+from the CNI.
+When this flag is set,
+the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag
+should be used when creating the socket
+to instruct libbpf not to load the default libbpf program on the netdev.
+Instead the loading is handled by the AF_XDP Device Plugin.
+
+
+Limitations
+-----------
+
+For DPDK versions <= v23.11 the Unix Domain Socket file path
+appears in the pod at "/tmp/afxdp.sock".
+The handshake implementation in the AF_XDP PMD
+is only compatible with the AF_XDP Device Plugin up to commit id `38317c2`_
+and the pod is limited to a single netdev.
+
+.. note::
+
+ DPDK AF_XDP PMD <= v23.11 will not work with the latest version
+ of the AF_XDP Device Plugin.
+
+The issue is if a single pod requests different devices from different pools,
+it results in multiple UDS servers serving the pod
+with the container using only a single mount point for their UDS as ``/tmp/afxdp.sock``.
+This means that at best one device might be able to complete the handshake.
+This has been fixed in the AF_XDP Device Plugin so that the mount point in the pods
+for the UDS appear at ``/tmp/afxdp_dp/<netdev>/afxdp.sock``.
+Later versions of DPDK fix this hardcoded path in the PMD
+alongside the ``use_cni`` parameter.
+
+.. _38317c2: https://github.com/intel/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669
+
+
+Prerequisites
+-------------
+
+Device Plugin and DPDK container prerequisites:
+
+* Create a DPDK container image.
+
+* Set up the device plugin and prepare the Pod Spec as described in
+ the instructions for `AF_XDP Device Plugin for Kubernetes`_.
+
+* The Docker image should contain the libbpf and libxdp libraries,
+ which are dependencies for AF_XDP,
+ and should include support for the ``ethtool`` command.
+
+* The Pod should have enabled the capabilities
+ ``CAP_NET_RAW`` for AF_XDP socket creation,
+ ``IPC_LOCK`` for umem creation and
+ ``CAP_BPF`` (for Kernel < 5.19) along with support for hugepages.
+
+ .. note::
+
+ For Kernel versions < 5.19, all BPF sys calls required CAP_BPF,
+ to access maps shared between the eBFP program and the userspace program.
+ Kernels >= 5.19, only requires CAP_BPF for map creation (BPF_MAP_CREATE)
+ and loading programs (BPF_PROG_LOAD).
+
+* Increase locked memory limit so containers have enough memory for packet buffers.
+ For example:
+
+ .. code-block:: console
+
+ cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf
+ [Service]
+ LimitMEMLOCK=infinity
+ EOF
+
+* dpdk-testpmd application should have AF_XDP feature enabled.
+
+ For further information see the docs for the: :doc:`../../nics/af_xdp`.
+
+
+Example
+-------
+
+Build a DPDK container image (using Docker)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Create a Dockerfile (should be placed in top level DPDK directory):
+
+ .. code-block:: console
+
+ FROM fedora:38
+
+ # Setup container to build DPDK applications
+ RUN dnf -y upgrade && dnf -y install \
+ libbsd-devel \
+ numactl-libs \
+ libbpf-devel \
+ libbpf \
+ meson \
+ ninja-build \
+ libxdp-devel \
+ libxdp \
+ numactl-devel \
+ python3-pyelftools \
+ python38 \
+ iproute
+ RUN dnf groupinstall -y 'Development Tools'
+
+ # Create DPDK dir and copy over sources
+ # Create DPDK dir and copy over sources
+ COPY ./ /dpdk
+ WORKDIR /dpdk
+
+ # Build DPDK
+ RUN meson setup build
+ RUN ninja -C build
+
+2. Build a DPDK container image (using Docker)
+
+ .. code-block:: console
+
+ # docker build -t dpdk -f Dockerfile
+
+Run dpdk-testpmd with the AF_XDP Device Plugin + CNI
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Clone the AF_XDP Device plugin and CNI
+
+ .. code-block:: console
+
+ # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git
+
+ .. note::
+
+ Ensure you have the AF_XDP Device Plugin + CNI prerequisites installed.
+
+* Build the AF_XDP Device plugin and CNI
+
+ .. code-block:: console
+
+ # cd afxdp-plugins-for-kubernetes/
+ # make image
+
+* Make sure to modify the image used by the `daemonset.yml`_ file
+ in the deployments directory with the following configuration:
+
+ .. _daemonset.yml : https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/deployments/daemonset.yml
+
+ .. code-block:: yaml
+
+ image: afxdp-device-plugin:latest
+
+ .. note::
+
+ This will select the AF_XDP DP image that was built locally.
+ Detailed configuration options can be found in the AF_XDP Device Plugin `readme`_ .
+
+ .. _readme: https://github.com/intel/afxdp-plugins-for-kubernetes#readme
+
+* Deploy the AF_XDP Device Plugin and CNI
+
+ .. code-block:: console
+
+ # kubectl create -f deployments/daemonset.yml
+
+* Create the Network Attachment definition
+
+ .. code-block:: console
+
+ # kubectl create -f nad.yaml
+
+ Sample nad.yml
+
+ .. code-block:: yaml
+
+ apiVersion: "k8s.cni.cncf.io/v1"
+ kind: NetworkAttachmentDefinition
+ metadata:
+ name: afxdp-network
+ annotations:
+ k8s.v1.cni.cncf.io/resourceName: afxdp/myPool
+ spec:
+ config: '{
+ "cniVersion": "0.3.0",
+ "type": "afxdp",
+ "mode": "primary",
+ "logFile": "afxdp-cni.log",
+ "logLevel": "debug",
+ "ethtoolCmds" : ["-N -device- rx-flow-hash udp4 fn",
+ "-N -device- flow-type udp4 dst-port 2152 action 22"
+ ],
+ "ipam": {
+ "type": "host-local",
+ "subnet": "192.168.1.0/24",
+ "rangeStart": "192.168.1.200",
+ "rangeEnd": "192.168.1.220",
+ "routes": [
+ { "dst": "0.0.0.0/0" }
+ ],
+ "gateway": "192.168.1.1"
+ }
+ }'
+
+ For further reference please use the example provided by the AF_XDP DP `nad.yaml`_
+
+ .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/examples/network-attachment-definition.yaml
+
+* Run the Pod
+
+ .. code-block:: console
+
+ # kubectl create -f pod.yaml
+
+ Sample pod.yaml:
+
+ .. code-block:: yaml
+
+ apiVersion: v1
+ kind: Pod
+ metadata:
+ name: dpdk
+ annotations:
+ k8s.v1.cni.cncf.io/networks: afxdp-network
+ spec:
+ containers:
+ - name: testpmd
+ image: dpdk:latest
+ command: ["tail", "-f", "/dev/null"]
+ securityContext:
+ capabilities:
+ add:
+ - NET_RAW
+ - IPC_LOCK
+ resources:
+ requests:
+ afxdp/myPool: '1'
+ limits:
+ hugepages-1Gi: 2Gi
+ cpu: 2
+ memory: 256Mi
+ afxdp/myPool: '1'
+ volumeMounts:
+ - name: hugepages
+ mountPath: /dev/hugepages
+ volumes:
+ - name: hugepages
+ emptyDir:
+ medium: HugePages
+
+ For further reference please use the `pod.yaml`_
+
+ .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/examples/pod-spec.yaml
+
+* Run DPDK with a command like the following:
+
+ .. code-block:: console
+
+ kubectl exec -i <Pod name> --container <containers name> -- \
+ /<Path>/dpdk-testpmd -l 0,1 --no-pci \
+ --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \
+ --no-mlockall --in-memory \
+ -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index 71a3381c36..a7692e8a97 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -8,7 +8,7 @@ HowTo Guides
:maxdepth: 2
:numbered:
- af_xdp_cni
+ af_xdp_dp
lm_bond_virtio_sriov
lm_virtio_vhost_user
flow_bifurcation
diff --git a/doc/guides/nics/af_xdp.rst b/doc/guides/nics/af_xdp.rst
index 1932525d4d..4dd9c73742 100644
--- a/doc/guides/nics/af_xdp.rst
+++ b/doc/guides/nics/af_xdp.rst
@@ -155,9 +155,9 @@ use_cni
~~~~~~~
The EAL vdev argument ``use_cni`` is used to indicate that the user wishes to
-enable the `AF_XDP Plugin for Kubernetes`_ within a DPDK application.
+enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod.
-.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
+.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
.. code-block:: console
--
2.34.1
---
Diff of the applied patch vs upstream commit (please double-check if non-empty:
---
--- - 2024-07-12 18:40:15.022579617 +0800
+++ 0016-doc-fix-AF_XDP-device-plugin-howto.patch 2024-07-12 18:40:13.946594246 +0800
@@ -1 +1 @@
-From 696982d5efd1ac40576c076b4358a144181d4a0e Mon Sep 17 00:00:00 2001
+From 190a4d4844aeca2c94e4b0f49065057928bf202e Mon Sep 17 00:00:00 2001
@@ -4,0 +5,3 @@
+Cc: Xueming Li <xuemingl at nvidia.com>
+
+[ upstream commit 696982d5efd1ac40576c076b4358a144181d4a0e ]
@@ -13 +15,0 @@
-Cc: stable at dpdk.org
@@ -28 +30 @@
-index 237698aa52..de326949c4 100644
+index f3b429b747..c6bf385a59 100644
@@ -31 +33 @@
-@@ -907,7 +907,7 @@ Martin Klozik <martinx.klozik at intel.com>
+@@ -899,7 +899,7 @@ Martin Klozik <martinx.klozik at intel.com>
More information about the stable
mailing list