[dpdk-dev] [PATCH v3 5/6] virtio: Add support for qtest virtio-net PMD
Tetsuya Mukawa
mukawa at igel.co.jp
Fri Mar 4 10:53:32 CET 2016
On 2016/03/04 15:10, Tan, Jianfeng wrote:
> Hi Tetsuya,
>
> On 3/4/2016 1:05 PM, Tetsuya Mukawa wrote:
>> On 2016/03/04 11:18, Tan, Jianfeng wrote:
>>> Hi Tetsuya,
>>>
>>> Seems that this patch is too long. Is it possible to split into
>>> multiple commits?
>> Hi Jianfeng,
>>
>> Sure, will do.
>>
>>> On 2/22/2016 4:17 PM, Tetsuya Mukawa wrote:
>>>> The patch adds a new virtio-net PMD configuration that allows the
>>>> PMD to
>>>> work on host as if the PMD is in VM.
>>>> Here is new configuration for virtio-net PMD.
>>>> - CONFIG_RTE_VIRTIO_VDEV_QTEST
>>>> To use this mode, EAL needs map all hugepages as one file. Also the
>>>> file
>>>> should be mapped between (1 << 31) and (1 << 44). And start address
>>>> should be aligned by EAL memory size.
>>>>
>>>> To allocate like above, use below options.
>>>> --single-file
>>>> --range-virtaddr=0x80000000-0x100000000000
>>>> --align-memsize
>>>> If a free regions isn't found, EAL will return error.
>>>>
>>>> To prepare virtio-net device on host, the users need to invoke QEMU
>>>> process in special qtest mode. This mode is mainly used for testing
>>>> QEMU
>>>> devices from outer process. In this mode, no guest runs.
>>>> Here is QEMU command line.
>>>>
>>>> $ qemu-system-x86_64 \
>>>> -machine pc-i440fx-1.4,accel=qtest \
>>>> -display none -qtest-log /dev/null \
>>>> -qtest unix:/tmp/socket,server \
>>>> -netdev type=tap,script=/etc/qemu-ifup,id=net0,queues=1 \
>>>> -device
>>>> virtio-net-pci,netdev=net0,mq=on,disable-modern=false,addr=3 \
>>>> -chardev socket,id=chr1,path=/tmp/ivshmem,server \
>>>> -device ivshmem,size=1G,chardev=chr1,vectors=1,addr=4
>>>>
>>>> * Should use qemu-2.5.1, or above.
>>>> * QEMU process is needed per port.
>>>> * virtio-1.0 device are only supported.
>>>> * The vhost backends like vhost-net and vhost-user can be
>>>> specified.
>>>> * In most cases, just using above command is enough, but you can
>>>> also
>>>> specify other QEMU virtio-net options.
>>>> * Only checked "pc-i440fx-1.4" machine, but may work with other
>>>> machines.
>>>> * Should not add "--enable-kvm" to QEMU command line.
>>> Correct me if wrong: all control msgs go through qemu process, e.g.,
>>> tx notifications and rx interrupts need follow frontend-qemu-backend
>>> path. Question: qemu is started without --enable-kvm, as I understand,
>>> ioeventfd, the basis of kickfd/callfd, will not be available. So how
>>> does qemu kick backend or be kicked by backend?
>> Actually, vhost-backend process will receive kickfd and callfd as -1.
>> (Currently, we have a bug in librte_vhost, because the library treats -1
>> as "not initialized state". But actually without "--enable-kvm", -1 will
>> be set by qemu to initialize kickfd and callfd. I will send a patch for
>> the issue with next patch series.)
>
> Yes, we noticed the problem too: librte_vhost judges virtio_is_ready
> by whether both fds are set. But except that, what's kernel's way to
> do the judgement? In addition, it would be better to be a independent
> fix patch.
>
Even if fd is -1, "VHOST_SET_VRING_KICK/CALL" will be issued.
So we can know correct timing.
I have tested current QEMU, and reviewed kernel code to know how kernel
treats it.
And I've fount below.
- vhost-user backend case.
"-1" will be set as kickfd and callfd.
- kernel vhost-net backend case.
Actually eventfd will be set as kickfd and callfd.
So if backend is vhost-net, vhost-net will use eventfd.
>>
>> In our case, virtio-net driver and vhost-backend driver are PMD. So we
>> don't use kickfd and callfd, right?
>>
>> If you worried about vhost-net case, vhost-net kernel thread will work
>> without ioeventfd and irqfd.
>> In this case, virtio-net PMD can kick the vhost-net by accessing
>> VIRTIO_PCI_QUEUE_NOTIFY register.
>> (vhost-net doesn't need to kick virtio-net driver, because the driver is
>> PMD.)
>
> I ask this question because I think interrupt mode will help the
> scalability. Return to your solution, by accessing
> VIRTIO_PCI_QUEUE_NOTIFY register, virtio-net PMD can only wake up
> qemu, but how does qemu wakes up vhost-net under the case that kickfd
> = callfd = -1.
>
Answer is above.
It seems QEMU will set eventfd for vhost-net case.
Only with vhost-user backend case, QEMU sends "-1" as kickfd and callfd.
>>>> After invoking QEMU, the PMD can connect to QEMU process using unix
>>>> domain sockets. Over these sockets, virtio-net, ivshmem and piix3
>>>> device in QEMU are probed by the PMD.
>>>> Here is example of command line.
>>>>
>>>> $ testpmd -c f -n 1 -m 1024 --no-pci --single-file \
>>>> --range-virtaddr=0x80000000-0x100000000000 --align-memsize \
>>>>
>>>> --vdev="eth_qtest_virtio0,qtest=/tmp/socket,ivshmem=/tmp/ivshmem"\
>>>> -- --disable-hw-vlan --txqflags=0xf00 -i
>>>>
>>>> Please specify same unix domain sockets and memory size in both QEMU
>>>> and DPDK command lines like above.
>>>> The share memory size should be power of 2, because ivshmem only
>>>> accepts such memory size.
>>>>
>>>> Signed-off-by: Tetsuya Mukawa <mukawa at igel.co.jp>
>>>> ---
>>>> config/common_linuxapp | 1 +
>>>> drivers/net/virtio/Makefile | 4 +
>>>> drivers/net/virtio/qtest.c | 1342
>>>> ++++++++++++++++++++++++++++++++++++
>>>> drivers/net/virtio/qtest.h | 65 ++
>>>> drivers/net/virtio/virtio_ethdev.c | 383 +++++++++-
>>>> drivers/net/virtio/virtio_pci.c | 364 +++++++++-
>>>> drivers/net/virtio/virtio_pci.h | 5 +-
>>>> 7 files changed, 2122 insertions(+), 42 deletions(-)
>>>> create mode 100644 drivers/net/virtio/qtest.c
>>>> create mode 100644 drivers/net/virtio/qtest.h
>>>>
>>>> diff --git a/config/common_linuxapp b/config/common_linuxapp
>>>> index 452f39c..f6e53bc 100644
>>>> --- a/config/common_linuxapp
>>>> +++ b/config/common_linuxapp
>>>> @@ -533,3 +533,4 @@ CONFIG_RTE_TEST_PMD_RECORD_BURST_STATS=n
>>>> # Enable virtio support for container
>>>> #
>>>> CONFIG_RTE_VIRTIO_VDEV=y
>>>> +CONFIG_RTE_VIRTIO_VDEV_QTEST=y
>>>> diff --git a/drivers/net/virtio/Makefile b/drivers/net/virtio/Makefile
>>>> index ef920f9..6c11378 100644
>>>> --- a/drivers/net/virtio/Makefile
>>>> +++ b/drivers/net/virtio/Makefile
>>>> @@ -56,6 +56,10 @@ ifeq ($(CONFIG_RTE_VIRTIO_VDEV),y)
>>>> SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += vhost_embedded.c
>>>> endif
>>>> +ifeq ($(CONFIG_RTE_VIRTIO_VDEV_QTEST),y)
>>>> + SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += qtest.c
>>>> +endif
>>>> +
>>>> # this lib depends upon:
>>>> DEPDIRS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += lib/librte_eal
>>>> lib/librte_ether
>>>> DEPDIRS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += lib/librte_mempool
>>>> lib/librte_mbuf
>>>> diff --git a/drivers/net/virtio/qtest.c b/drivers/net/virtio/qtest.c
>>>> new file mode 100644
>>>> index 0000000..061aab5
>>>> --- /dev/null
>>>> +++ b/drivers/net/virtio/qtest.c
>>>> @@ -0,0 +1,1342 @@
>>>> +/*-
>>>> + * BSD LICENSE
>>>> + *
>>>> + * Copyright(c) 2016 IGEL Co., Ltd. All rights reserved.
>>>> + * All rights reserved.
>>>> + *
>>>> + * Redistribution and use in source and binary forms, with or
>>>> without
>>>> + * modification, are permitted provided that the following
>>>> conditions
>>>> + * are met:
>>>> + *
>>>> + * * Redistributions of source code must retain the above
>>>> copyright
>>>> + * notice, this list of conditions and the following
>>>> disclaimer.
>>>> + * * Redistributions in binary form must reproduce the above
>>>> copyright
>>>> + * notice, this list of conditions and the following
>>>> disclaimer in
>>>> + * the documentation and/or other materials provided with the
>>>> + * distribution.
>>>> + * * Neither the name of IGEL Co., Ltd. nor the names of its
>>>> + * contributors may be used to endorse or promote products
>>>> derived
>>>> + * from this software without specific prior written
>>>> permission.
>>>> + *
>>>> + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
>>>> CONTRIBUTORS
>>>> + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
>>>> NOT
>>>> + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
>>>> FITNESS FOR
>>>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
>>>> COPYRIGHT
>>>> + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
>>>> INCIDENTAL,
>>>> + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
>>>> + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
>>>> OF USE,
>>>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
>>>> ON ANY
>>>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
>>>> TORT
>>>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>>>> THE USE
>>>> + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
>>>> DAMAGE.
>>>> + */
>>>> +#include <stdint.h>
>>>> +#include <stdlib.h>
>>>> +#include <string.h>
>>>> +#include <unistd.h>
>>>> +#include <sys/types.h>
>>>> +#include <sys/socket.h>
>>>> +#include <sys/un.h>
>>>> +#include <sys/queue.h>
>>>> +#include <signal.h>
>>>> +#include <pthread.h>
>>>> +#include <sys/stat.h>
>>>> +#include <fcntl.h>
>>>> +#include <sys/eventfd.h>
>>>> +#include <linux/pci_regs.h>
>>>> +
>>>> +#include <rte_memory.h>
>>>> +#include <rte_malloc.h>
>>>> +#include <rte_common.h>
>>>> +#include <rte_interrupts.h>
>>>> +
>>>> +#include "virtio_pci.h"
>>>> +#include "virtio_logs.h"
>>>> +#include "virtio_ethdev.h"
>>>> +#include "qtest.h"
>>>> +
>>>> +#define NB_BAR 6
>>>> +
>>>> +/* PIIX3 configuration registers */
>>>> +#define PIIX3_REG_ADDR_PIRQA 0x60
>>>> +#define PIIX3_REG_ADDR_PIRQB 0x61
>>>> +#define PIIX3_REG_ADDR_PIRQC 0x62
>>>> +#define PIIX3_REG_ADDR_PIRQD 0x63
>>>> +
>>>> +/* Device information */
>>>> +#define VIRTIO_NET_DEVICE_ID 0x1000
>>>> +#define VIRTIO_NET_VENDOR_ID 0x1af4
>>>> +#define VIRTIO_NET_IRQ_NUM 10
>>>> +#define IVSHMEM_DEVICE_ID 0x1110
>>>> +#define IVSHMEM_VENDOR_ID 0x1af4
>>>> +#define IVSHMEM_PROTOCOL_VERSION 0
>>>> +#define PIIX3_DEVICE_ID 0x7000
>>>> +#define PIIX3_VENDOR_ID 0x8086
>>>> +
>>>> +/* ------------------------------------------------------------
>>>> + * IO port mapping of qtest guest
>>>> + * ------------------------------------------------------------
>>>> + * 0x0000 - 0xbfff : not used
>>>> + * 0xc000 - 0xc03f : virtio-net(BAR0)
>>>> + * 0xc040 - 0xffff : not used
>>>> + *
>>>> + * ------------------------------------------------------------
>>>> + * Memory mapping of qtest quest
>>>> + * ------------------------------------------------------------
>>>> + * 0x00000000_00000000 - 0x00000000_3fffffff : not used
>>>> + * 0x00000000_40000000 - 0x00000000_40000fff : virtio-net(BAR1)
>>>> + * 0x00000000_40001000 - 0x00000000_40ffffff : not used
>>>> + * 0x00000000_41000000 - 0x00000000_417fffff : virtio-net(BAR4)
>>>> + * 0x00000000_41800000 - 0x00000000_41ffffff : not used
>>>> + * 0x00000000_42000000 - 0x00000000_420000ff : ivshmem(BAR0)
>>>> + * 0x00000000_42000100 - 0x00000000_42ffffff : not used
>>>> + * 0x00000000_80000000 - 0xffffffff_ffffffff : ivshmem(BAR2)
>>>> + *
>>> Is it possible to arrange multiple virtio-net devices here? What's the
>>> challenges?
>> Yes, you can manage multiple virtio-net devices here, if you define
>> correct memory map.
>
> So this memory map will be decided here or in qemu?
Here, we can decide memory map of "all PCI devices".
And we will initialize them along with this memory mapping.
Actually some memory mapping is in QEMU also.
For example, first 1MB is allocated for firmware. We should avoid to
overlap it.
It seems this is decided in machine setting file.
Actually not well documented, but please see "hw/i386/pc.c" of QEMU code.
>
>>
>>> Seems that lots of below code do the same work as libqos. So can we
>>> just link libqos? Or we need to maintain this code.
>> Problem is libqos will be GPL.
>> So I wrote the code from scratch.
>
> OK, great! To make it extensible for other virtio devices (like scsi),
> we need to abstract those code as a qtest utils, and virtio-net
> related code into other .c file.
>
OK, while splitting my long patch, I will separate qtest utils from
virtio-net related code.
If someone uses this extension for other QEMU devices, actually it's
very interesting!
Thanks,
Tetsuya
More information about the dev
mailing list