[dpdk-dev] [PATCH v7 00/19] vhost: add postcopy live-migration support
Maxime Coquelin
maxime.coquelin at redhat.com
Tue Oct 16 10:37:53 CEST 2018
On 10/12/2018 02:40 PM, Maxime Coquelin wrote:
> In this v7:
> - Move vh_result enum declaration to vhost.h (Ilya)
> - Remove check in read_fd_message() for more fds received
> than requested as already checked with MSG_CTRUNC (Ilya)
> - Don't return an error in vhost-crypto post message handler
> if not a crypto message as all messages go through it.
>
> With classic live-migration, the VM runs on source while its
> content is being migrated to destination. When pages already
> migrated to destination are dirtied by the source, they get
> copied until both source and destination memory converge.
> At that time, the source is stopped and destination is
> started.
>
> With postcopy live-migration, the VM is started on destination
> before all the memory has been migrated. When the VM tries to
> access a page that haven't been migrated yet, a pagefault is
> triggered, handled by userfaultfd which pauses the thread.
> A Qemu thread in charge of postcopy request the source for
> the missing page. Once received and mapped, the paused thread
> gets resumed.
>
> Userfaultfd supports handling faults from a different process,
> and Qemu supports postcopy with vhost-user backends since
> v2.12.
>
> One problem encountered with classic live-migration for VMs
> relying on vhost-user backends is that when the traffic is
> high (e.g. PVP), it happens that it never converges as
> pages gets dirtied at a faster rate than they are copied
> to the destination.
> It is expected this problem sould be solved with using
> postcopy, as rings memory and buffers will be copied once,
> when destination will pagefault on them.
>
> Note that it will certainly require a rebase to apply on top
> of Nikolay's vhost-user message handling rework.
>
> Steps to test postcopy:
> 1. Run DPDK's Testpmd application on source:
> ./install/bin/testpmd -m 512 --file-prefix=src -l 0,2 -n 4 \
> --vdev 'net_vhost0,iface=/tmp/vu-src' -- --portmask=1 -i \
> --rxq=1 --txq=1 --nb-cores=1 --eth-peer=0,52:54:00:11:22:12 \
> --no-mlockall
>
> 2. Run DPDK's Testpmd application on destination:
> ./install/bin/testpmd -m 512 --file-prefix=dst -l 0,2 -n 4 \
> --vdev 'net_vhost0,iface=/tmp/vu-dst,postcopy-support=1' -- --portmask=1 -i \
> --rxq=1 --txq=1 --nb-cores=1 --eth-peer=0,52:54:00:11:22:12 \
> --no-mlockall
>
> 3. Launch VM on source:
> ./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 3G -smp 2 -cpu host \
> -object memory-backend-file,id=mem,size=3G,mem-path=/dev/shm,share=on \
> -numa node,memdev=mem -mem-prealloc \
> -chardev socket,id=char0,path=/tmp/vu-src \
> -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
> -device virtio-net-pci,netdev=mynet1 /home/virt/rhel7.6-1-clone.qcow2 \
> -net none -vnc :0 -monitor stdio
>
> 4. Launch VM on destination:
> ./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 3G -smp 2 -cpu host \
> -object memory-backend-file,id=mem,size=3G,mem-path=/dev/shm,share=on \
> -numa node,memdev=mem -mem-prealloc \
> -chardev socket,id=char0,path=/tmp/vu-dst \
> -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
> -device virtio-net-pci,netdev=mynet1 /home/virt/rhel7.6-1-clone.qcow2 \
> -net none -vnc :1 -monitor stdio -incoming tcp::8888
>
> 5. In both testpmd prompts, start flooding the virtio-net device:
> testpmd> set fwd txonly
> testpmd> start
>
> 6. In destination's Qemu monitor, enable postcopy:
> (qemu) migrate_set_capability postcopy-ram on
>
> 7. In source's Qemu monitor, enable postcopy and launch migration:
> (qemu) migrate_set_capability postcopy-ram on
> (qemu) migrate -d tcp:0:8888
> (qemu) migrate_start_postcopy
>
> Maxime Coquelin (19):
> vhost: fix messages results handling
> vhost: fix return code of messages requiring replies
> vhost: clarify reply-ack in case a reply was already sent
> vhost: fix payload size of reply
> vhost: fix error handling when mem table gets updated
> vhost: define postcopy protocol flag
> vhost: add number of fds to vhost-user messages and use it
> vhost: pass socket fd to message handling callbacks
> vhost: enable fds passing when sending vhost-user messages
> vhost: add config flag for postcopy feature
> vhost: introduce postcopy's advise message
> vhost: add support for postcopy's listen message
> vhost: register new regions with userfaultfd
> vhost: avoid useless VhostUserMemory copy
> vhost: send userfault range addresses back to qemu
> vhost: add support to postcopy's end request
> vhost: restrict postcopy live-migration enablement
> net/vhost: add parameter to enable postcopy support
> vhost: enable postcopy protocol feature
>
> config/common_linuxapp | 1 +
> doc/guides/nics/vhost.rst | 5 +
> doc/guides/prog_guide/vhost_lib.rst | 8 +
> drivers/net/vhost/rte_eth_vhost.c | 13 ++
> lib/librte_vhost/meson.build | 2 +
> lib/librte_vhost/rte_vhost.h | 5 +
> lib/librte_vhost/socket.c | 46 +++-
> lib/librte_vhost/vhost.h | 32 ++-
> lib/librte_vhost/vhost_crypto.c | 25 +-
> lib/librte_vhost/vhost_user.c | 344 +++++++++++++++++++++-------
> lib/librte_vhost/vhost_user.h | 12 +-
> 11 files changed, 377 insertions(+), 116 deletions(-)
>
Applied to dpdk-next-virtio/master.
Maxime
More information about the dev
mailing list