[PATCH 00/21] Add control queue & MQ support to Virtio-user vDPA

[Maxime Coquelin]

Oups, forgot to set subject prefix to v3.

On 2/9/23 10:16, Maxime Coquelin wrote:
> This series introduces control queue support for Vhost-vDPA
> backend. This is a requirement to support multiqueue, but
> be usefull for other features like RSS for example.
> 
> Since the Virtio-user layer of the Virtio PMD must handle
> some control messages, like the number of queue pairs to
> be used by the device, a shadow control queue is created
> at Virtio-user layer.
> 
> Control messages from the regular Virtio control queue
> are still dequeues and handled if needed by the Virtio-user
> layer, and are then forwarded to the shadow control queue
> so that the physical vDPA device can handle them.
> 
> This model is similar to the one adopted by the QEMU
> project.
> 
> In order to avoid code duplication, virtqueue allocation
> and control queue message sending has been factored out
> of the Virtio layer to be reusable by the Virtio-user
> layer.
> 
> Finally, in order to support vDPA hardware which may
> support large number of queues, last patch removes the
> 8 queue pairs limitation by dynamically allocating
> vring metadata.
> 
> The series has been tested with Nvidia Cx-6 DX NIC
> with up to 16 queue pairs:
> 
> # echo 0 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs
> # echo 0 > /sys/bus/pci/devices/0000\:3b\:00.1/sriov_numvfs
> # modprobe vhost_vdpa
> # modprobe mlx5_vdpa
> # echo 1 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs
> # echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/unbind
> # devlink dev eswitch set pci/0000:3b:00.0 mode switchdev
> # echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/bind
> # vdpa dev add name vdpa0 mgmtdev pci/0000:3b:00.2 mac 00:11:22:33:44:03  max_vqp 16
> # ulimit -l unlimited
> # dpdk-testpmd -l 0,2,4,6 --socket-mem 1024,0  --vdev 'virtio_user0,path=/dev/vhost-vdpa-0' --no-pci -n 3 -- --nb-cores=3 -i --rxq=16 --txq=16
> 
> Changes in v3:
> ==============
> - Trivial code simplifications (Eugenio)
> 
> Changes in v2:
> ==============
> - Fix double spaces (Chenbo)
> - Get rid of uneeded gotos (Stephen)
> - Only allocate packed ring metadata if supported (Chenbo)
> - Rebased on top of main
> 
> Maxime Coquelin (21):
>    net/virtio: move CVQ code into a dedicated file
>    net/virtio: introduce notify callback for control queue
>    net/virtio: virtqueue headers alloc refactoring
>    net/virtio: remove port ID info from Rx queue
>    net/virtio: remove unused fields in Tx queue struct
>    net/virtio: remove unused queue ID field in Rx queue
>    net/virtio: remove unused Port ID in control queue
>    net/virtio: move vring memzone to virtqueue struct
>    net/virtio: refactor indirect desc headers init
>    net/virtio: alloc Rx SW ring only if vectorized path
>    net/virtio: extract virtqueue init from virtio queue init
>    net/virtio-user: fix device starting failure handling
>    net/virtio-user: simplify queues setup
>    net/virtio-user: use proper type for number of queue pairs
>    net/virtio-user: get max number of queue pairs from device
>    net/virtio-user: allocate shadow control queue
>    net/virtio-user: send shadow virtqueue info to the backend
>    net/virtio-user: add new callback to enable control queue
>    net/virtio-user: forward control messages to shadow queue
>    net/virtio-user: advertize control VQ support with vDPA
>    net/virtio-user: remove max queues limitation
> 
>   drivers/net/virtio/meson.build                |   1 +
>   drivers/net/virtio/virtio.h                   |   6 -
>   drivers/net/virtio/virtio_cvq.c               | 229 +++++++++
>   drivers/net/virtio/virtio_cvq.h               | 127 +++++
>   drivers/net/virtio/virtio_ethdev.c            | 472 +-----------------
>   drivers/net/virtio/virtio_rxtx.c              |  47 +-
>   drivers/net/virtio/virtio_rxtx.h              |  31 +-
>   drivers/net/virtio/virtio_rxtx_packed.c       |   3 +-
>   drivers/net/virtio/virtio_rxtx_simple.c       |   3 +-
>   drivers/net/virtio/virtio_rxtx_simple.h       |   7 +-
>   .../net/virtio/virtio_rxtx_simple_altivec.c   |   4 +-
>   drivers/net/virtio/virtio_rxtx_simple_neon.c  |   4 +-
>   drivers/net/virtio/virtio_rxtx_simple_sse.c   |   4 +-
>   drivers/net/virtio/virtio_user/vhost.h        |   1 +
>   drivers/net/virtio/virtio_user/vhost_vdpa.c   |  16 +-
>   .../net/virtio/virtio_user/virtio_user_dev.c  | 305 +++++++++--
>   .../net/virtio/virtio_user/virtio_user_dev.h  |  30 +-
>   drivers/net/virtio/virtio_user_ethdev.c       |  49 +-
>   drivers/net/virtio/virtqueue.c                | 346 ++++++++++++-
>   drivers/net/virtio/virtqueue.h                | 127 +----
>   20 files changed, 1066 insertions(+), 746 deletions(-)
>   create mode 100644 drivers/net/virtio/virtio_cvq.c
>   create mode 100644 drivers/net/virtio/virtio_cvq.h
>