[dpdk-dev] [PATCH v2 00/17] vhost: add postcopy live-migration support
Maxime Coquelin
maxime.coquelin at redhat.com
Tue Oct 2 11:36:34 CEST 2018
In this v2:
- Rebase on top of Nikolay message handling series. It
requires passing an extra parameter to message handlers (
the fd of the socket as set_mem_table needs to send an
intermediate reply).
- Preliminary patches to fix issues with message handling
rework not handling replies properly.
- Handle userfaultfd region registration errors properly.
- Don't build postcopy by default as userfaultd only
landed in v4.3 kernel. It gets automatically enabled with
Meson if headers are present.
With classic live-migration, the VM runs on source while its
content is being migrated to destination. When pages already
migrated to destination are dirtied by the source, they get
copied until both source and destination memory converge.
At that time, the source is stopped and destination is
started.
With postcopy live-migration, the VM is started on destination
before all the memory has been migrated. When the VM tries to
access a page that haven't been migrated yet, a pagefault is
triggered, handled by userfaultfd which pauses the thread.
A Qemu thread in charge of postcopy request the source for
the missing page. Once received and mapped, the paused thread
gets resumed.
Userfaultfd supports handling faults from a different process,
and Qemu supports postcopy with vhost-user backends since
v2.12.
One problem encountered with classic live-migration for VMs
relying on vhost-user backends is that when the traffic is
high (e.g. PVP), it happens that it never converges as
pages gets dirtied at a faster rate than they are copied
to the destination.
It is expected this problem sould be solved with using
postcopy, as rings memory and buffers will be copied once,
when destination will pagefault on them.
Note that it will certainly require a rebase to apply on top
of Nikolay's vhost-user message handling rework.
Steps to test postcopy:
1. Run DPDK's Testpmd application on source:
./install/bin/testpmd -m 512 --file-prefix=src -l 0,2 -n 4 \
--vdev 'net_vhost0,iface=/tmp/vu-src' -- --portmask=1 -i \
--rxq=1 --txq=1 --nb-cores=1 --eth-peer=0,52:54:00:11:22:12 \
--no-mlockall
2. Run DPDK's Testpmd application on destination:
./install/bin/testpmd -m 512 --file-prefix=dst -l 0,2 -n 4 \
--vdev 'net_vhost0,iface=/tmp/vu-dst,postcopy-support=1' -- --portmask=1 -i \
--rxq=1 --txq=1 --nb-cores=1 --eth-peer=0,52:54:00:11:22:12 \
--no-mlockall
3. Launch VM on source:
./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 3G -smp 2 -cpu host \
-object memory-backend-file,id=mem,size=3G,mem-path=/dev/shm,share=on \
-numa node,memdev=mem -mem-prealloc \
-chardev socket,id=char0,path=/tmp/vu-src \
-netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
-device virtio-net-pci,netdev=mynet1 /home/virt/rhel7.6-1-clone.qcow2 \
-net none -vnc :0 -monitor stdio
4. Launch VM on destination:
./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 3G -smp 2 -cpu host \
-object memory-backend-file,id=mem,size=3G,mem-path=/dev/shm,share=on \
-numa node,memdev=mem -mem-prealloc \
-chardev socket,id=char0,path=/tmp/vu-dst \
-netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
-device virtio-net-pci,netdev=mynet1 /home/virt/rhel7.6-1-clone.qcow2 \
-net none -vnc :1 -monitor stdio -incoming tcp::8888
5. In both testpmd prompts, start flooding the virtio-net device:
testpmd> set fwd txonly
testpmd> start
6. In destination's Qemu monitor, enable postcopy:
(qemu) migrate_set_capability postcopy-ram on
7. In source's Qemu monitor, enable postcopy and launch migration:
(qemu) migrate_set_capability postcopy-ram on
(qemu) migrate -d tcp:0:8888
(qemu) migrate_start_postcopy
Maxime Coquelin (17):
vhost: fix messages error checks
vhost: fix return code of messages requiring replies
vhost: fix error handling when mem table gets updated
vhost: define postcopy protocol flag
vhost: add number of fds to vhost-user messages and use it
vhost: pass socket fd to message handling callbacks
vhost: enable fds passing when sending vhost-user messages
vhost: add config flag for postcopy feature
vhost: introduce postcopy's advise message
vhost: add support for postcopy's listen message
vhost: register new regions with userfaultfd
vhost: avoid useless VhostUserMemory copy
vhost: send userfault range addresses back to qemu
vhost: add support to postcopy's end request
vhost: enable postcopy protocol feature
vhost: add flag to enable postcopy live-migration
net/vhost: add parameter to enable postcopy support
config/common_linuxapp | 1 +
doc/guides/nics/vhost.rst | 5 +
doc/guides/prog_guide/vhost_lib.rst | 8 +
drivers/net/vhost/rte_eth_vhost.c | 13 ++
lib/librte_vhost/meson.build | 2 +
lib/librte_vhost/rte_vhost.h | 5 +
lib/librte_vhost/socket.c | 40 +++-
lib/librte_vhost/vhost.h | 3 +
lib/librte_vhost/vhost_user.c | 304 +++++++++++++++++++++++-----
lib/librte_vhost/vhost_user.h | 12 +-
10 files changed, 327 insertions(+), 66 deletions(-)
--
2.17.1
More information about the dev
mailing list