[dpdk-dev] [PATCH v4 0/8] virtio_user as an alternative exception path

[Jianfeng Tan]

v4:
  - Fix a clang compiling error by removing "NULL" line in the definition
    of vhost_msg_strings. This error does not show up when it's defined
    as a static variable, so not necessary to fix it in stable branch.
  - Query kernel to get how many regions are supported, default 64 regions.
  - Set TUNSETSNDBUF to INT_MAX.
  - When get_features, unmask those backend-specific feature bits.
  - Remove VHOST_KERNEL_MAX_QUEUES (8) restriction, but due to another
    restriction by VIRTIO_MAX_VIRTQUEUES (8), we still cannot configure
    more than 8 queues.
  - Add a howto document.

v3:
  - Drop the patch to postpone driver ok sending patch, superseded it
    with a bug fix to disable all virtqueues and re-init the device.
    (you might wonder why not just send reset owner msg. Under my test,
     it causes spinlock deadlock problem when killing the program).
  - Avoid compiling error on 32-bit system for pointer convert.
  - Fix a bug in patch "abstract virtio user backend ops", vhostfd is
    not properly assigned.
  - Fix a "MQ cannot be used" bug in v2, which is related to strip
    some feature bits that vhost kernel does not recognize.
  - Update release note.

v2: (Lots of them are from yuanhan's comment)
  - Add offloding feature.
  - Add multiqueue support.
  - Add a new patch to postpone the sending of driver ok notification.
  - Put fix patch ahead of the whole patch series.
  - Split original 0001 patch into 0003 and 0004 patches.
  - Remove the original vhost_internal design, just add those into
    struct virtio_user_dev for simplicity.
  - Reword "control" to "send_request".
  - Reword "host_features" to "device_features". 

In v16.07, we upstreamed a virtual device, virtio_user (with vhost-user
as the backend). The path to go with a vhost-kernel backend has been
dropped for bad performance comparing to vhost-user and code simplicity.

But after a second thought, virtio_user + vhost-kernel is a good 
candidate as an exceptional path, such as KNI, which exchanges packets
with kernel networking stack.
  - maintenance: vhost-net (kernel) is upstreamed and extensively used 
    kernel module. We don't need any out-of-tree module like KNI.
  - performance: as with KNI, this solution would use one or more
    kthreads to send/receive packets from user space DPDK applications,
    which has little impact on user space polling thread (except that
    it might enter into kernel space to wake up those kthreads if
    necessary).
  - features: vhost-net is born to be a networking solution, which has
    lots of networking related featuers, like multi queue, tso, multi-seg
    mbuf, etc.


How to test:

Here is a simple test case for:
                                _______testpmd_______
    iperf-c/iperf-s -- tap0 -- [virtio_user0 -- ixgbe]
                                                  |
                                                  |
                                                  |
                             iperf-s/iperf-c -- ixgbe (kernel driver, eth3)

Step 1: Remove MAC addres setting in testpmd csum fwd engine.

    diff --git a/app/test-pmd/csumonly.c b/app/test-pmd/csumonly.c
    index 57e6ae2..225c7a8 100644
    --- a/app/test-pmd/csumonly.c
    +++ b/app/test-pmd/csumonly.c
    @@ -706,10 +706,12 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
                     * and inner headers */
     
                    eth_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
    +#if 0
                    ether_addr_copy(&peer_eth_addrs[fs->peer_addr],
                                    &eth_hdr->d_addr);
                    ether_addr_copy(&ports[fs->tx_port].eth_addr,
                                    &eth_hdr->s_addr);
    +#endif
                    parse_ethernet(eth_hdr, &info);
                    l3_hdr = (char *)eth_hdr + info.l2_len;
     
Step 2: bind one ixgbe with igb_uio, and start testpmd
    $(testpmd) -c 0xc -n 4 \
        --vdev=virtio_user0,path=/dev/vhost-net,queue_size=1024 \
        -- -i --txqflags=0x0 --disable-hw-vlan --enable-lro \
        --enable-rx-cksum --rxd=1024 --txd=1024

To test multiqueue, start testpmd like this:
    $(testpmd) -c 0xc -n 4 \
        --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 \
        -- -i --txqflags=0x0 --disable-hw-vlan --enable-lro --enable-rx-cksum \
        --txq=2 --rxq=2 --rxd=1024 --txd=1024

Step 3: start testpmd: (port 0 is ixgbe, port 1 is virtio_user)
    # set fwd csum
    # csum set ip hw 0
    # csum set tcp hw 0
    # csum set ip sw 1
    # csum set tcp hw 1
    # tso set 1448 0
    # tso set 1448 1
    # start

Step 4: start the other end with below script
    $ ip netns add ns1
    $ ip link set eth3 netns ns1
    $ ip netns exec ns1 ifconfig eth3 1.1.1.2/24 up
    $ ip netns exec ns1 taskset 0xf0 iperf3 -s -i 1

Use below command if you change the position of iperf-c and iperf-s. 
$ ip netns exec ns1 numactl -N 0 iperf3 -c 1.1.1.3 -i 1 -t 30

Step 5: up the tap0 and configure ip
    $ ifconfig tap0 1.1.1.3/24 up

Step 6: start test
    $ iperf3 -c 1.1.1.2 -i 1 -t 30

Signed-off-by: Jianfeng Tan <jianfeng.tan at intel.com>

Jianfeng Tan (8):
  net/virtio_user: fix wrongly get/set features
  net/virtio_user: fix not properly reset device
  net/virtio_user: move vhost user specific code
  net/virtio_user: abstract virtio user backend ops
  net/virtio_user: add vhost kernel support
  net/virtio_user: enable offloading
  net/virtio_user: enable multiqueue with vhost kernel
  doc: add guide to use virtio_user as exceptional path

 .../img/virtio_user_as_excpetional_path.png        | Bin 0 -> 38600 bytes
 doc/guides/prog_guide/index.rst                    |   1 +
 .../prog_guide/virtio_user_as_exceptional_path.rst | 104 ++++++
 doc/guides/rel_notes/release_17_02.rst             |  20 +
 drivers/net/virtio/Makefile                        |   2 +
 drivers/net/virtio/virtio_ethdev.h                 |   5 +
 drivers/net/virtio/virtio_user/vhost.h             |  51 +--
 drivers/net/virtio/virtio_user/vhost_kernel.c      | 401 +++++++++++++++++++++
 drivers/net/virtio/virtio_user/vhost_kernel_tap.c  | 133 +++++++
 drivers/net/virtio/virtio_user/vhost_kernel_tap.h  |  67 ++++
 drivers/net/virtio/virtio_user/vhost_user.c        |  98 +++--
 drivers/net/virtio/virtio_user/virtio_user_dev.c   | 153 +++++---
 drivers/net/virtio/virtio_user/virtio_user_dev.h   |  15 +-
 drivers/net/virtio/virtio_user_ethdev.c            |  20 +-
 14 files changed, 945 insertions(+), 125 deletions(-)
 create mode 100644 doc/guides/prog_guide/img/virtio_user_as_excpetional_path.png
 create mode 100644 doc/guides/prog_guide/virtio_user_as_exceptional_path.rst
 create mode 100644 drivers/net/virtio/virtio_user/vhost_kernel.c
 create mode 100644 drivers/net/virtio/virtio_user/vhost_kernel_tap.c
 create mode 100644 drivers/net/virtio/virtio_user/vhost_kernel_tap.h

-- 
2.7.4