[dpdk-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Wed, Feb 01, 2017 at 11:41:19AM +0000, Daniel P. Berrange wrote:
> On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote:
> > 
> > 
> > On 02/01/2017 10:43 AM, Daniel P. Berrange wrote:
> > > On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
> > > > On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
> > > 
> > > > > Solution 3: Libvirt queries OVS for vhost backend version string: *OK*
> > > > > ======================================================================
> > > > > 
> > > > > 
> > > > >  The idea is to have a table of supported versions, associated to
> > > > > key/value pairs. Libvirt could query the list of supported versions
> > > > > strings for each hosts, and select the first common one among all hosts.
> > > > 
> > > > How does libvirt know what hosts to ask? Libvirt aims on managing a
> > > > single host. It has no knowledge of other hosts on the network. That's
> > > > task for upper layers like RHEV, OpenStack, etc.
> > > > 
> > > > > 
> > > > >  Then, libvirt would ask OVS to probe the vhost-user interfaces in the
> > > > > selected version (compatibility mode). For example host A runs OVS-2.7,
> > > > > and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode
> > > > > (e.g. with indirect descriptors disabled), which should be selected at
> > > > > vhost-user interface probe time.
> > > > > 
> > > > >  Advantage of doing so is that libvirt does not need any update if new
> > > > > keys are introduced (i.e. it does not need to know how the new keys have
> > > > > to be handled), all these checks remain in OVS's vhost-user implementation.
> > > > 
> > > > And that's where they should stay. Duplicating code between projects
> > > > will inevitably lead to a divergence.
> > > > 
> > > > > 
> > > > >  Ideally, we would support per vhost-user interface compatibility mode,
> > > > > which may have an impact also on DPDK API, as the Virtio feature update
> > > > > API is global, and not per port.
> > > > 
> > > > In general, I don't think we want any kind of this logic in libvirt. Either:
> > > > 
> > > > a) fallback logic should be implemented in qemu (e.g. upon migration it
> > > > should detect that the migrated guest uses certain version and thus set
> > > > backend to use that version or error out and cancel migration), or
> > > > 
> > > > b) libvirt would grew another element/attribute to specify version of
> > > > vhost-user backend in use and do nothing more than pass it to qemu. At
> > > > the same time, we can provide an API (or extend and existing one, e.g.
> > > > virsh domcapabilities) to list all available versions on given host.
> > > > Upper layer, which knows what are the possible hosts suitable for
> > > > virtualization, can then use this API to ask all the hosts, construct
> > > > the matrix, select preferred version and put it into libvirt's domain XML.
> > > > 
> > > > But frankly, I don't like b) that much. Lets put the fact this is OVS
> > > > aside for a moment. Just pretend this is a generic device in qemu. Would
> > > > we do the same magic with it? No! Or lets talk about machine types. You
> > > > spawn -M type$((X+1)) guest and then decide to migrate it to a host with
> > > > older qemu wich supports just typeX. Well, you get an error. Do we care?
> > > > Not at all! It's your responsibility (as user/admin) to upgrade the qemu
> > > > so that it supports new machine type. I think the same applies to OVS.
> > > 
> > > With machine types, if the latest machine type is X, libvirt allows
> > > the mgmt app to spawn a guest with mcahine type X-1, so that you can
> > > later migrate the VM to a host with older QEMU.
> > > 
> > > With the vhost user device, the VM will always be launched with version
> > > Y. There's currently no way to request launching the vhost user device
> > > wtih version Y-1. So even if you set your machine type to X-1 for
> > > compat with older host, vhost user will be stuck at version Y preventing
> > > the migration.
> > > 
> > > One argument would be to say that the vhost user featureset should be
> > > determined by the machine type version, instead of introducing a new
> > > version. The complexity is that vhost-user is a pretty dumb device
> > > from QEMUs POV - most of the interesting logic & the features that
> > > need to be constrained lie in code outside of QEMU, in whatever
> > > server is connected to the vhost user socket.
> > > 
> > > So I can see the value in allowing a simple version string to be
> > > associated with the vhost-user NIC.
> > > 
> > > What I'm unclear about is how we would be able to report capabilities
> > > for a host to enumerate what versions were possible. Libvirt doesn't
> > > interact with any of the 3rd party vhost user servers, so it is probably
> > > out of scope - it'd be upto the higher level mgmt app to talk to DPDK
> > > and figure out what versions it supports.
> > > 
> > > That does make me wonder though if libvirt & QEMU need to be involved
> > > at all in any part.
> > 
> > Indeed, if the higher level management tool decides for the VM's machine
> > type, it is where it should also be done for the vhost-user backend. I
> > now understand this does not make much sense to have libvirt being
> > involved in this, all (querying/selecting/setting compat mode) should be
> > manageable in the upper layer.
> > 
> > I'm not familiar with these layers, so your inputs are really helpful.
> > 
> > > 
> > > When provisioning a new guest, the mgmt app presumably has to talk to
> > > DPDK to setup the NIC port, so DPDK is ready when QEMU launches and
> > > connects. Surely as part of that NIC port setup, it could directly
> > > tell DPDK which version or featureset to permit on the port ? It is
> > > not obvious why the version string has to be fed in via QEMU.
> > No it is not, my proposal was that libvirt set the version string in
> > OVS, QEMU was not involved.
> > 
> > From these inputs, the idea remains valid, except that libvirt is not
> > the right place to manage this. Instead, RHEV, Openstack or any other
> > management tool should handle the compat mode selection.
> 
> It depends where / how in OVS it needs to be set. The only stuff libvirt
> does with OVS is to run 'add-port' and 'del-port' commands via the ovs
> cli tool. We pass through arguments from the port profile stored in the
> XML config.
> 
>   <interface type='bridge'>
>     <source bridge='ovsbr'/>
>     <virtualport type='openvswitch'>
>       <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
>     </virtualport>
>   </interface>
> 
> eg those things in <parameters/> get passed as cli args to the 'add-port'
> command. Soo if add-port needs this new version string, then we'd need
> to add the version to the openvswitch virtualport XML.
> 
> If the version is provided to OVS in a different command, then it would
> probably be outside scope of libvirt.
> 
> Regards,
> Daniel

OK but how does it end up there in the 1st place?
Something like we do for qemu, where there's a query
for "pc" alias, and that is stored, seems called for.

Further, it seems ugly to assume add will fail
if it can not support what is required.
Better to query what is supported I think.

> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|