[dpdk-users] rte_eth_dev_rss_reta_update appears to have no effect with net_ixgbe driver

[Williams, Christopher]

We're using receive side scaling with Intel NICs that use the net_ixgbe PMD.

Our application needs to be able to follow entire conversations in a higher-level protocol, such as HTTP or SMTP, which requires us to set a hashing key such that (a, b) hashes to the same value as (b, a) (i.e. client->server packets and server->client packets must end up in the same queue). The requirement to be able to follow FTP conversations limits us to hashing over IP addresses only, since any FTP conversation will use multiple TCP ports.

Hashing IP addresses only results in extremely uneven distribution of packets into the rx queues. We *should* be able to fix this by setting a redirection table to bias packets towards one table rather than another. However, configuring a redirection table appears to have no effect.

The following C++ function is a slight simplification of what we're actually doing:

#define MAX_RETA_SIZE (ETH_RSS_RETA_SIZE_512 / RTE_RETA_GROUP_SIZE)

void config_eth_dev(uint8_t port_id, uint16_t nRx)
{
    rte_eth_conf eth_config;
    memset(&eth_config, 0, sizeof(eth_config));
    struct rte_eth_dev_info inf;
    rte_eth_dev_info_get(port_id, &inf);
    eth_config.rxmode.mq_mode = ETH_MQ_RX_RSS; // Enable receive-side scaling
    // Set a hashing key such that frames from a->b and b->a are inserted into
    // the same hardware queue
    static uint8_t hashKey[] = {
        0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
        0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
        0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
        0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
        0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
    };
    eth_config.rx_adv_conf.rss_conf.rss_key = hashKey;
    eth_config.rx_adv_conf.rss_conf.rss_key_len = sizeof(hashKey);
    // Hash IP addresses only so that, e.g., FTP packets go to the same rx queue
    eth_config.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IPV4 | ETH_RSS_IPV6;


    if (rte_eth_dev_configure(port_id, nRx, 1, &eth_config))
    {
        throw std::runtime_error("rte_eth_dev_configure");
    }

    rte_eth_dev_info_get(port_id, &inf);
    // Set a redirection table - this should distribute traffic more-or-less evenly
    // to each queue
    uint16_t rSize = inf.reta_size;
    // RX queue distribution for two queues
    static uint16_t dist2[] = {
        0, 0, 0, 0, 1, 1, 1,
    };
    // RX queue distribution for four queues
    static uint16_t dist4[] = {
        0, 0, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3,
    };
    uint16_t *pd = nullptr, i;
    size_t nd = 0;
    switch (nRx) {
        case 4:
            pd = dist4;
            nd = sizeof(dist4) / sizeof(uint16_t);
            break;
        case 2:
            pd = dist2;
            nd = sizeof(dist2) / sizeof(uint16_t);
            break;
        default:
            break;
    }
    if (rSize && rSize <= ETH_RSS_RETA_SIZE_512 && pd)
    {
        struct rte_eth_rss_reta_entry64 rConf[MAX_RETA_SIZE];
        memset(rConf, 0, sizeof(rConf));
        for (i = 0; i < rSize; i++)
        {
            rConf[i / RTE_RETA_GROUP_SIZE].mask = UINT64_MAX;
        }

        for (i = 0; i < rSize; i++)
        {
            uint16_t reta_id = i / RTE_RETA_GROUP_SIZE;
            uint16_t reta_pos = i % RTE_RETA_GROUP_SIZE;
            uint16_t rss_qs_pos = i % nd;

            rConf[reta_id].reta[reta_pos] = pd[rss_qs_pos];
        }
        if (rte_eth_dev_rss_reta_update(port_id, rConf, rSize))
        {
            throw std::runtime_error("rte_eth_dev_rss_reta_update");
        }
    }
}

The code for configuring the redirection table is lifted from the ip_pipeline sample application.

Is there something we're doing wrong here (or something we're failing to do)?

Christopher Williams.



________________________________

McAfee Security UK Limited is registered in England and Wales with its registered address at C/O Skadden, Arps, Slate, Meagher & Flom (UK) LLP, 40 Bank Street, Canary Wharf, London, United Kingdom, E14 5DS, Company No. 10472868