[dpdk-dev] [PATCH v5 6/6] doc/guides/: provide IOAT sample app guide

Bruce Richardson bruce.richardson at intel.com
Fri Sep 27 13:01:30 CEST 2019


On Fri, Sep 20, 2019 at 09:37:14AM +0200, Marcin Baran wrote:
> Added guide for IOAT sample app usage and
> code description.
> 
> Signed-off-by: Marcin Baran <marcinx.baran at intel.com>
> ---
>  doc/guides/sample_app_ug/index.rst |   1 +
>  doc/guides/sample_app_ug/intro.rst |   4 +
>  doc/guides/sample_app_ug/ioat.rst  | 764 +++++++++++++++++++++++++++++
>  3 files changed, 769 insertions(+)
>  create mode 100644 doc/guides/sample_app_ug/ioat.rst
> 
> diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
> index f23f8f59e..a6a1d9e7a 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -23,6 +23,7 @@ Sample Applications User Guides
>      ip_reassembly
>      kernel_nic_interface
>      keep_alive
> +    ioat
>      l2_forward_crypto
>      l2_forward_job_stats
>      l2_forward_real_virtual
> diff --git a/doc/guides/sample_app_ug/intro.rst b/doc/guides/sample_app_ug/intro.rst
> index 90704194a..74462312f 100644
> --- a/doc/guides/sample_app_ug/intro.rst
> +++ b/doc/guides/sample_app_ug/intro.rst
> @@ -91,6 +91,10 @@ examples are highlighted below.
>    forwarding, or ``l3fwd`` application does forwarding based on Internet
>    Protocol, IPv4 or IPv6 like a simple router.
>  
> +* :doc:`Hardware packet copying<ioat>`: The Hardware packet copying,
> +  or ``ioatfwd`` application demonstrates how to use IOAT rawdev driver for
> +  copying packets between two threads.
> +
>  * :doc:`Packet Distributor<dist_app>`: The Packet Distributor
>    demonstrates how to distribute packets arriving on an Rx port to different
>    cores for processing and transmission.
> diff --git a/doc/guides/sample_app_ug/ioat.rst b/doc/guides/sample_app_ug/ioat.rst
> new file mode 100644
> index 000000000..69621673b
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/ioat.rst
> @@ -0,0 +1,764 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2019 Intel Corporation.
> +
> +Sample Application of packet copying using Intel\ |reg| QuickData Technology
> +============================================================================
> +
> +Overview
> +--------
> +
> +This sample is intended as a demonstration of the basic components of a DPDK
> +forwarding application and example of how to use IOAT driver API to make
> +packets copies.
> +
> +Also while forwarding, the MAC addresses are affected as follows:
> +
> +*   The source MAC address is replaced by the TX port MAC address
> +
> +*   The destination MAC address is replaced by  02:00:00:00:00:TX_PORT_ID
> +
> +This application can be used to compare performance of using software packet
> +copy with copy done using a DMA device for different sizes of packets.
> +The example will print out statistics each second. The stats shows
> +received/send packets and packets dropped or failed to copy.
> +
> +Compiling the Application
> +-------------------------
> +
> +To compile the sample application see :doc:`compiling`.
> +
> +The application is located in the ``ioat`` sub-directory.
> +
> +
> +Running the Application
> +-----------------------
> +
> +In order to run the hardware copy application, the copying device
> +needs to be bound to user-space IO driver.
> +
> +Refer to the *IOAT Rawdev Driver for Intel\ |reg| QuickData Technology*
> +guide for information on using the driver.
> +

The document is not called that, as the IOAT guide is just part of the
overall rawdev document. So I suggest you just reference the rawdev guide.

> +The application requires a number of command line options:
> +
> +.. code-block:: console
> +
> +    ./build/ioatfwd [EAL options] -- -p MASK [-q NQ] [-s RS] [-c <sw|hw>]
> +        [--[no-]mac-updating]
> +
> +where,
> +
> +*   p MASK: A hexadecimal bitmask of the ports to configure

Is this a mandatory parameter, or does the app use all detected ports by
default, e.g. like testpmd?

> +
> +*   q NQ: Number of Rx queues used per port equivalent to CBDMA channels
> +    per port
> +
> +*   c CT: Performed packet copy type: software (sw) or hardware using
> +    DMA (hw)

What is the default? Same for next two parameters.

> +
> +*   s RS: Size of IOAT rawdev ring for hardware copy mode or rte_ring for
> +    software copy mode
> +
> +*   --[no-]mac-updating: Whether MAC address of packets should be changed
> +    or not
> +
> +The application can be launched in various configurations depending on
> +provided parameters. Each port can use up to 2 lcores: one of them receives

The app uses 2 data plane cores, total, rather than 2 per-port, I believe.
It would be good to explain the difference here that with 2 cores the
copies are done on one core, and the mac updates on the second one.

> +incoming traffic and makes a copy of each packet. The second lcore then
> +updates MAC address and sends the copy. If one lcore per port is used,
> +both operations are done sequentially. For each configuration an additional
> +lcore is needed since master lcore in use which is responsible for

... since the master lcore does not handle traffic but is responsible for

> +configuration, statistics printing and safe deinitialization of all ports
> +and devices.

s/deinitialization/shutdown/

> +
> +The application can use a maximum of 8 ports.

Is this a hard limit in the app, if so explain why. I see the stats
arrays are limited by "RTE_MAX_ETHPORTS".

> +
> +To run the application in a Linux environment with 3 lcores (one of them
> +is master lcore), 1 port (port 0), software copying and MAC updating issue
> +the command:

s/1 port/a single port/

s/one of them is master lcore/the master lcore, plus two forwarding cores/

Similar comments would apply to text immediately below too.

> +
> +.. code-block:: console
> +
> +    $ ./build/ioatfwd -l 0-2 -n 2 -- -p 0x1 --mac-updating -c sw
> +
> +To run the application in a Linux environment with 2 lcores (one of them
> +is master lcore), 2 ports (ports 0 and 1), hardware copying and no MAC
> +updating issue the command:
> +
> +.. code-block:: console
> +
> +    $ ./build/ioatfwd -l 0-1 -n 1 -- -p 0x3 --no-mac-updating -c hw
> +
> +Refer to the *DPDK Getting Started Guide* for general information on
> +running applications and the Environment Abstraction Layer (EAL) options.
> +
> +Explanation
> +-----------
> +
> +The following sections provide an explanation of the main components of the
> +code.
> +
> +All DPDK library functions used in the sample code are prefixed with
> +``rte_`` and are explained in detail in the *DPDK API Documentation*.
> +
> +
> +The Main Function
> +~~~~~~~~~~~~~~~~~
> +
> +The ``main()`` function performs the initialization and calls the execution
> +threads for each lcore.
> +
> +The first task is to initialize the Environment Abstraction Layer (EAL).
> +The ``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()``
> +function. The value returned is the number of parsed arguments:
> +
> +.. code-block:: c
> +
> +    /* init EAL */
> +    ret = rte_eal_init(argc, argv);
> +    if (ret < 0)
> +        rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n");
> +
> +
> +The ``main()`` also allocates a mempool to hold the mbufs (Message Buffers)
> +used by the application:
> +
> +.. code-block:: c
> +
> +    nb_mbufs = RTE_MAX(rte_eth_dev_count_avail() * (nb_rxd + nb_txd
> +        + MAX_PKT_BURST + rte_lcore_count() * MEMPOOL_CACHE_SIZE),
> +        MIN_POOL_SIZE);
> +
> +    /* Create the mbuf pool */
> +    ioat_pktmbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", nb_mbufs,
> +        MEMPOOL_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE,
> +        rte_socket_id());
> +    if (ioat_pktmbuf_pool == NULL)
> +        rte_exit(EXIT_FAILURE, "Cannot init mbuf pool\n");
> +
> +Mbufs are the packet buffer structure used by DPDK. They are explained in
> +detail in the "Mbuf Library" section of the *DPDK Programmer's Guide*.
> +
> +The ``main()`` function also initializes the ports:
> +
> +.. code-block:: c
> +
> +    /* Initialise each port */
> +    RTE_ETH_FOREACH_DEV(portid) {
> +        port_init(portid, ioat_pktmbuf_pool);
> +    }
> +
> +Each port is configured using ``port_init()``:
> +
> +.. code-block:: c
> +
> +     /*
> +     * Initializes a given port using global settings and with the RX buffers
> +     * coming from the mbuf_pool passed as a parameter.
> +     */
> +    static inline void
> +    port_init(uint16_t portid, struct rte_mempool *mbuf_pool, uint16_t nb_queues)
> +    {
> +        /* configuring port to use RSS for multiple RX queues */
> +        static const struct rte_eth_conf port_conf = {
> +            .rxmode = {
> +                .mq_mode        = ETH_MQ_RX_RSS,
> +                .max_rx_pkt_len = RTE_ETHER_MAX_LEN
> +            },
> +            .rx_adv_conf = {
> +                .rss_conf = {
> +                    .rss_key = NULL,
> +                    .rss_hf = ETH_RSS_PROTO_MASK,
> +                }
> +            }
> +        };
> +
> +        struct rte_eth_rxconf rxq_conf;
> +        struct rte_eth_txconf txq_conf;
> +        struct rte_eth_conf local_port_conf = port_conf;
> +        struct rte_eth_dev_info dev_info;
> +        int ret, i;
> +
> +        /* Skip ports that are not enabled */
> +        if ((ioat_enabled_port_mask & (1 << portid)) == 0) {
> +            printf("Skipping disabled port %u\n", portid);
> +            return;
> +        }
> +
> +        /* Init port */
> +        printf("Initializing port %u... ", portid);
> +        fflush(stdout);
> +        rte_eth_dev_info_get(portid, &dev_info);
> +        local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
> +            dev_info.flow_type_rss_offloads;
> +        if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
> +            local_port_conf.txmode.offloads |=
> +                DEV_TX_OFFLOAD_MBUF_FAST_FREE;
> +        ret = rte_eth_dev_configure(portid, nb_queues, 1, &local_port_conf);
> +        if (ret < 0)
> +            rte_exit(EXIT_FAILURE, "Cannot configure device:"
> +                " err=%d, port=%u\n", ret, portid);
> +
> +        ret = rte_eth_dev_adjust_nb_rx_tx_desc(portid, &nb_rxd,
> +                            &nb_txd);
> +        if (ret < 0)
> +            rte_exit(EXIT_FAILURE,
> +                "Cannot adjust number of descriptors: err=%d, port=%u\n",
> +                ret, portid);
> +
> +        rte_eth_macaddr_get(portid, &ioat_ports_eth_addr[portid]);
> +
> +        /* Init Rx queues */
> +        rxq_conf = dev_info.default_rxconf;
> +        rxq_conf.offloads = local_port_conf.rxmode.offloads;
> +        for (i = 0; i < nb_queues; i++) {
> +            ret = rte_eth_rx_queue_setup(portid, i, nb_rxd,
> +                rte_eth_dev_socket_id(portid), &rxq_conf,
> +                mbuf_pool);
> +            if (ret < 0)
> +                rte_exit(EXIT_FAILURE,
> +                    "rte_eth_rx_queue_setup:err=%d,port=%u, queue_id=%u\n",
> +                    ret, portid, i);
> +        }
> +
> +        /* Init one TX queue on each port */
> +        txq_conf = dev_info.default_txconf;
> +        txq_conf.offloads = local_port_conf.txmode.offloads;
> +        ret = rte_eth_tx_queue_setup(portid, 0, nb_txd,
> +                rte_eth_dev_socket_id(portid),
> +                &txq_conf);
> +        if (ret < 0)
> +            rte_exit(EXIT_FAILURE,
> +                "rte_eth_tx_queue_setup:err=%d,port=%u\n",
> +                ret, portid);
> +
> +        /* Initialize TX buffers */
> +        tx_buffer[portid] = rte_zmalloc_socket("tx_buffer",
> +                RTE_ETH_TX_BUFFER_SIZE(MAX_PKT_BURST), 0,
> +                rte_eth_dev_socket_id(portid));
> +        if (tx_buffer[portid] == NULL)
> +            rte_exit(EXIT_FAILURE,
> +                "Cannot allocate buffer for tx on port %u\n",
> +                portid);
> +
> +        rte_eth_tx_buffer_init(tx_buffer[portid], MAX_PKT_BURST);
> +
> +        ret = rte_eth_tx_buffer_set_err_callback(tx_buffer[portid],
> +                rte_eth_tx_buffer_count_callback,
> +                &port_statistics.tx_dropped[portid]);
> +        if (ret < 0)
> +            rte_exit(EXIT_FAILURE,
> +                "Cannot set error callback for tx buffer on port %u\n",
> +                portid);
> +
> +        /* Start device */
> +        ret = rte_eth_dev_start(portid);
> +        if (ret < 0)
> +            rte_exit(EXIT_FAILURE,
> +                "rte_eth_dev_start:err=%d, port=%u\n",
> +                ret, portid);
> +
> +        rte_eth_promiscuous_enable(portid);
> +
> +        printf("Port %u, MAC address: %02X:%02X:%02X:%02X:%02X:%02X\n\n",
> +                portid,
> +                ioat_ports_eth_addr[portid].addr_bytes[0],
> +                ioat_ports_eth_addr[portid].addr_bytes[1],
> +                ioat_ports_eth_addr[portid].addr_bytes[2],
> +                ioat_ports_eth_addr[portid].addr_bytes[3],
> +                ioat_ports_eth_addr[portid].addr_bytes[4],
> +                ioat_ports_eth_addr[portid].addr_bytes[5]);
> +
> +        cfg.ports[cfg.nb_ports].rxtx_port = portid;
> +        cfg.ports[cfg.nb_ports++].nb_queues = nb_queues;
> +    }
> +

This code is probably quite similar to that in other sample apps, so I
don't think we need to include the full function here. It makes updating
the code more difficult, so just refer to the function as doing the port
init and leave it at that, I think. The snippets below give enough detail.

> +The Ethernet ports are configured with local settings using the
> +``rte_eth_dev_configure()`` function and the ``port_conf`` struct.
> +The RSS is enabled so that multiple Rx queues could be used for
> +packet receiving and copying by multiple CBDMA channels per port:
> +
> +.. code-block:: c
> +
> +    /* configuring port to use RSS for multiple RX queues */
> +    static const struct rte_eth_conf port_conf = {
> +        .rxmode = {
> +            .mq_mode        = ETH_MQ_RX_RSS,
> +            .max_rx_pkt_len = RTE_ETHER_MAX_LEN
> +        },
> +        .rx_adv_conf = {
> +            .rss_conf = {
> +                .rss_key = NULL,
> +                .rss_hf = ETH_RSS_PROTO_MASK,
> +            }
> +        }
> +    };
> +
> +For this example the ports are set up with the number of Rx queues provided
> +with -q option and 1 Tx queue using the ``rte_eth_rx_queue_setup()``
> +and ``rte_eth_tx_queue_setup()`` functions.
> +
> +The Ethernet port is then started:
> +
> +.. code-block:: c
> +
> +    ret = rte_eth_dev_start(portid);
> +    if (ret < 0)
> +        rte_exit(EXIT_FAILURE, "rte_eth_dev_start:err=%d, port=%u\n",
> +            ret, portid);
> +
> +
> +Finally the Rx port is set in promiscuous mode:
> +
> +.. code-block:: c
> +
> +    rte_eth_promiscuous_enable(portid);
> +
> +
> +After that each port application assigns resources needed.
> +
> +.. code-block:: c
> +
> +    check_link_status(ioat_enabled_port_mask);
> +
> +    if (!cfg.nb_ports) {
> +        rte_exit(EXIT_FAILURE,
> +            "All available ports are disabled. Please set portmask.\n");
> +    }
> +
> +    /* Check if there is enough lcores for all ports. */
> +    cfg.nb_lcores = rte_lcore_count() - 1;
> +    if (cfg.nb_lcores < 1)
> +        rte_exit(EXIT_FAILURE,
> +            "There should be at least one slave lcore.\n");
> +
> +    ret = 0;
> +
> +    if (copy_mode == COPY_MODE_IOAT_NUM) {
> +        assign_rawdevs();
> +    } else /* copy_mode == COPY_MODE_SW_NUM */ {
> +        assign_rings();
> +    }
> +
> +A link status is checked of each port enabled by port mask
> +using ``check_link_status()`` function.
> +

I don't think this block needs to be covered. No need to go into everything
in detail, just focus on the key parts of the app that are unique to it,
i.e. the copying and passing mbufs between threads parts.

> +.. code-block:: c
> +
> +    /* check link status, return true if at least one port is up */
> +    static int
> +    check_link_status(uint32_t port_mask)
> +    {
> +        uint16_t portid;
> +        struct rte_eth_link link;
> +        int retval = 0;
> +
> +        printf("\nChecking link status\n");
> +        RTE_ETH_FOREACH_DEV(portid) {
> +            if ((port_mask & (1 << portid)) == 0)
> +                continue;
> +
> +            memset(&link, 0, sizeof(link));
> +            rte_eth_link_get(portid, &link);
> +
> +            /* Print link status */
> +            if (link.link_status) {
> +                printf(
> +                    "Port %d Link Up. Speed %u Mbps - %s\n",
> +                    portid, link.link_speed,
> +                    (link.link_duplex == ETH_LINK_FULL_DUPLEX) ?
> +                    ("full-duplex") : ("half-duplex\n"));
> +                retval = 1;
> +            } else
> +                printf("Port %d Link Down\n", portid);
> +        }
> +        return retval;
> +    }
> +



More information about the dev mailing list