[dpdk-dev] [PATCH v4 0/8] virtio_user as an alternative exception path

Jianfeng Tan jianfeng.tan at intel.com
Fri Jan 13 13:18:33 CET 2017

  - Fix a clang compiling error by removing "NULL" line in the definition
    of vhost_msg_strings. This error does not show up when it's defined
    as a static variable, so not necessary to fix it in stable branch.
  - Query kernel to get how many regions are supported, default 64 regions.
  - When get_features, unmask those backend-specific feature bits.
  - Remove VHOST_KERNEL_MAX_QUEUES (8) restriction, but due to another
    restriction by VIRTIO_MAX_VIRTQUEUES (8), we still cannot configure
    more than 8 queues.
  - Add a howto document.

  - Drop the patch to postpone driver ok sending patch, superseded it
    with a bug fix to disable all virtqueues and re-init the device.
    (you might wonder why not just send reset owner msg. Under my test,
     it causes spinlock deadlock problem when killing the program).
  - Avoid compiling error on 32-bit system for pointer convert.
  - Fix a bug in patch "abstract virtio user backend ops", vhostfd is
    not properly assigned.
  - Fix a "MQ cannot be used" bug in v2, which is related to strip
    some feature bits that vhost kernel does not recognize.
  - Update release note.

v2: (Lots of them are from yuanhan's comment)
  - Add offloding feature.
  - Add multiqueue support.
  - Add a new patch to postpone the sending of driver ok notification.
  - Put fix patch ahead of the whole patch series.
  - Split original 0001 patch into 0003 and 0004 patches.
  - Remove the original vhost_internal design, just add those into
    struct virtio_user_dev for simplicity.
  - Reword "control" to "send_request".
  - Reword "host_features" to "device_features". 

In v16.07, we upstreamed a virtual device, virtio_user (with vhost-user
as the backend). The path to go with a vhost-kernel backend has been
dropped for bad performance comparing to vhost-user and code simplicity.

But after a second thought, virtio_user + vhost-kernel is a good 
candidate as an exceptional path, such as KNI, which exchanges packets
with kernel networking stack.
  - maintenance: vhost-net (kernel) is upstreamed and extensively used 
    kernel module. We don't need any out-of-tree module like KNI.
  - performance: as with KNI, this solution would use one or more
    kthreads to send/receive packets from user space DPDK applications,
    which has little impact on user space polling thread (except that
    it might enter into kernel space to wake up those kthreads if
  - features: vhost-net is born to be a networking solution, which has
    lots of networking related featuers, like multi queue, tso, multi-seg
    mbuf, etc.

How to test:

Here is a simple test case for:
    iperf-c/iperf-s -- tap0 -- [virtio_user0 -- ixgbe]
                             iperf-s/iperf-c -- ixgbe (kernel driver, eth3)

Step 1: Remove MAC addres setting in testpmd csum fwd engine.

    diff --git a/app/test-pmd/csumonly.c b/app/test-pmd/csumonly.c
    index 57e6ae2..225c7a8 100644
    --- a/app/test-pmd/csumonly.c
    +++ b/app/test-pmd/csumonly.c
    @@ -706,10 +706,12 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
                     * and inner headers */
                    eth_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
    +#if 0
                    parse_ethernet(eth_hdr, &info);
                    l3_hdr = (char *)eth_hdr + info.l2_len;
Step 2: bind one ixgbe with igb_uio, and start testpmd
    $(testpmd) -c 0xc -n 4 \
        --vdev=virtio_user0,path=/dev/vhost-net,queue_size=1024 \
        -- -i --txqflags=0x0 --disable-hw-vlan --enable-lro \
        --enable-rx-cksum --rxd=1024 --txd=1024

To test multiqueue, start testpmd like this:
    $(testpmd) -c 0xc -n 4 \
        --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 \
        -- -i --txqflags=0x0 --disable-hw-vlan --enable-lro --enable-rx-cksum \
        --txq=2 --rxq=2 --rxd=1024 --txd=1024

Step 3: start testpmd: (port 0 is ixgbe, port 1 is virtio_user)
    # set fwd csum
    # csum set ip hw 0
    # csum set tcp hw 0
    # csum set ip sw 1
    # csum set tcp hw 1
    # tso set 1448 0
    # tso set 1448 1
    # start

Step 4: start the other end with below script
    $ ip netns add ns1
    $ ip link set eth3 netns ns1
    $ ip netns exec ns1 ifconfig eth3 up
    $ ip netns exec ns1 taskset 0xf0 iperf3 -s -i 1

Use below command if you change the position of iperf-c and iperf-s. 
$ ip netns exec ns1 numactl -N 0 iperf3 -c -i 1 -t 30

Step 5: up the tap0 and configure ip
    $ ifconfig tap0 up

Step 6: start test
    $ iperf3 -c -i 1 -t 30

Signed-off-by: Jianfeng Tan <jianfeng.tan at intel.com>

Jianfeng Tan (8):
  net/virtio_user: fix wrongly get/set features
  net/virtio_user: fix not properly reset device
  net/virtio_user: move vhost user specific code
  net/virtio_user: abstract virtio user backend ops
  net/virtio_user: add vhost kernel support
  net/virtio_user: enable offloading
  net/virtio_user: enable multiqueue with vhost kernel
  doc: add guide to use virtio_user as exceptional path

 .../img/virtio_user_as_excpetional_path.png        | Bin 0 -> 38600 bytes
 doc/guides/prog_guide/index.rst                    |   1 +
 .../prog_guide/virtio_user_as_exceptional_path.rst | 104 ++++++
 doc/guides/rel_notes/release_17_02.rst             |  20 +
 drivers/net/virtio/Makefile                        |   2 +
 drivers/net/virtio/virtio_ethdev.h                 |   5 +
 drivers/net/virtio/virtio_user/vhost.h             |  51 +--
 drivers/net/virtio/virtio_user/vhost_kernel.c      | 401 +++++++++++++++++++++
 drivers/net/virtio/virtio_user/vhost_kernel_tap.c  | 133 +++++++
 drivers/net/virtio/virtio_user/vhost_kernel_tap.h  |  67 ++++
 drivers/net/virtio/virtio_user/vhost_user.c        |  98 +++--
 drivers/net/virtio/virtio_user/virtio_user_dev.c   | 153 +++++---
 drivers/net/virtio/virtio_user/virtio_user_dev.h   |  15 +-
 drivers/net/virtio/virtio_user_ethdev.c            |  20 +-
 14 files changed, 945 insertions(+), 125 deletions(-)
 create mode 100644 doc/guides/prog_guide/img/virtio_user_as_excpetional_path.png
 create mode 100644 doc/guides/prog_guide/virtio_user_as_exceptional_path.rst
 create mode 100644 drivers/net/virtio/virtio_user/vhost_kernel.c
 create mode 100644 drivers/net/virtio/virtio_user/vhost_kernel_tap.c
 create mode 100644 drivers/net/virtio/virtio_user/vhost_kernel_tap.h


More information about the dev mailing list