[PATCH v16 0/5] Support add/remove memory region and get-max-slots

[pravin.bathija at dell.com]

From: Pravin M Bathija <pravin.bathija at dell.com>

This is version v16 of the patchset and it incorporates the
recommendations made by Stephen Hemminger.

Patch 4/5
- Fixed use-after-free in vhost_user_add_mem_reg(): reg pointed into
  dev->mem->regions[] but dev_invalidate_vrings() ->
  translate_ring_addresses() -> numa_realloc() can relocate dev->mem ;
  re-drive the region pointer from the refreshed dev before reading
  host_user_addr for the reply.
- Gated the ADD_MEM_REG mapping-address reply on dev->postcopy_listening
  per the vhost-user spec (reply is postcopy-only); return
  RTE_VHOST_MSG_RESULT_OK in non-postcopy mode,  matching SET_MEM_TABLE,
  to avoid desyncing the channel.

This implementation has been extensively tested by doing Read/Write I/O
from multiple instances of fio + libblkio (front-end) talking to
spdk/dpdk (back-end) based drives. Tested with qemu front-end talking to
dpdk testpmd (back-end) performing add/removal of memory regions. Also
tested post-copy live migration after doing add_memory_region.

Version Log:
Version v16 (Current version): Incorporate code review suggestions from
Stephen Hemminger.

Version v15: Incorporate review suggestions from Maxime Coquelin.
Patch 4/5
- Changed VHOST_USER_REM_MEM_REG handler declaration from
  accepts_fd=true to accepts_fd=false, as the remove request does not
  expect FDs in ancillary data.
- Removed all close_msg_fds(ctx) calls from vhost_user_rem_mem_reg(), no
  longer needed since the handler is declared as not accepting FDs.
- Removed validate_msg_fds(dev, ctx, 0) check from
  vhost_user_rem_mem_reg(), as FD validation is now handled generically
  by the framework.
- Added targeted IOTLB cache invalidation in vhost_user_rem_mem_reg()
  using vhost_user_iotlb_cache_remove() for the removed region's GPA
  range, instead of the nuclear iotlb_flush_all() used by set_mem_table.

Version v14: Incorporate code review suggestions from Stephen Hemminger
and Fengcheng Wen.
Changes from Fengcheng Wen review:
Patch 3/5
- Moved free_all_mem_regions() call sites in vhost_user_set_mem_table()
  from patch 4/5 to patch 3/5 so each commit compiles independently
Patch 4/5
- Renamed _dev_invalidate_vrings() to vhost_user_invalidate_vrings() to
  follow vhost naming convention
-  Added comment explaining *pdev propagation through
   translate_ring_addresses / numa_realloc()
- Reordered local variables in vhost_user_add_mem_reg() and
  vhost_user_rem_mem_reg() by descending line length
- Shortened overlap check variable names (current_region_guest_start/end
  --> cur_start/end, proposed_region_guest_start/end -> new_start/end)
- Fixed DMA error path in vhost_user_add_mem_reg(): added
  free_new_region_no_dma label so async_dma_map_region(false) is not
  called when the map itself failed.
Changes from Stephen Hemminger review:
Patch 4/5
- vhost_user_add_mem_reg() now constructs a reply with the back-end's
  host mapping address in userspace_addr and returns
  RTE_VHOST_MSG_RESULT_REPLY per the vhost-user spec
- Added validate_msg_fds(dev, ctx, 0) in vhost_user_rem_mem_reg() to
  reject malformed messages with unexpected file descriptors
- Dropped unnecessary (uint64_t) cast in vhost_user_get_max_mem_slots()

Version v13: Incorporate code review suggestions from Fengcheng Wen
Patch 2/5
Renamed VhostUserSingleMemReg to VhostUserMemRegMsg and memory_single
to memreg
Patches 3/5 and 4/5
Relocated function remove_guest_pages from patch 3/5 to 4/5

Version v12: Incorporate code review suggestions from Maxime Coquelin
and ai-code-review.
Patch 3/5
Refactored async_dma_map() to delegate to async_dma_map_region(),
eliminating code duplication between the two functions.
Restored original comments in async_dma_map_region() explaining why
ENODEV and EINVAL errors are ignored (these were stripped in v10)
Reverted unnecessary changes to vhost_user_postcopy_register() --
removed the host_user_addr == 0 checks and reg_msg_index indirection
that were added in  v10, since this function is only called from
vhost_user_set_mem_table() where regions are always contiguous.

Version v11: Incorporate code review suggestions from Stephen Hemminger.
Patch 4/5
Fix incomplete cleanup in vhost_user_add_mem_reg() when
vhost_user_mmap_region() fails after the mmap succeeds (e.g.
add_guest_pages() realloc failure) realloc failure). The error path now
calls remove_guest_pages() and free_mem_region() to undo the mapping
and stale guest-page entries, preventing a leaked mmap and slot reuse
corruption. The plain close(fd) path is kept for pre-mmap failures.

Version v10: Incorporate code review suggestions from Stephen Hemminger.
Patch 4/5
Moved dev_invalidate_vrings after free_mem_region, array compaction, and
nregions decrement. This ensures translate_ring_addresses only sees
surviving memory regions, preventing vring pointers from resolving into
a region that is about to be unmapped.

Version v9: Incorporate code review suggestions from Stephen Hemminger.
Patch 3/5
Restored max_guest_pages initial value to hardcoded 8 instead of
VHOST_MEMORY_MAX_NREGIONS, matching upstream semantics.
Patch 4/5
Added close(reg->fd) and reg->fd = -1 before goto close_msg_fds in the
mmap failure path to fix fd leak after fd was moved from ctx->fds[0].
Converted dev_invalidate_vrings from a plain function to a macro +
implementation function pair, accepting message ID as a parameter so
the static_assert reports the correct handler at each call site.
Updated dev_invalidate_vrings call in add_mem_reg to pass
VHOST_USER_ADD_MEM_REG as message ID.
Updated dev_invalidate_vrings call in rem_mem_reg to pass
VHOST_USER_REM_MEM_REG as message ID.

Version v8:  Incorporate code review suggestions from Stephen Hemminger.
rewrite async_dma_map_region function to iterate guest pages by host
address range matching
change function dev_invalidate_vrings to accept a double pointer to
propagate pointer updates
new function remove_guest_pages was added
add_mem_reg error path was narrowed to only clean up the single failed
region instead of destroting all existing regions

Version v7: Incorporate code review suggestions from Maxime Coquelin.
Add debug messages to vhost_postcopy_register function.

Version v6: Added the enablement of this feature as a final patch in
this patch-set and other code optimizations as suggested by Maxime
Coquelin.

Version v5: removed the patch that increased the number of memory regions
from 8 to 128. This will be submitted as a separate feature at a later
point after incorporating additional optimizations. Also includes code
optimizations as suggested by Feng Cheng Wen.

Version v4: code optimizations as suggested by Feng Cheng Wen.

Version v3: code optimizations as suggested by Maxime Coquelin
and Thomas Monjalon.

Version v2: code optimizations as suggested by Maxime Coquelin.

Version v1: Initial patch set.

Pravin M Bathija (5):
  vhost: add user to mailmap and define to vhost hdr
  vhost: header defines for add/rem mem region
  vhost: refactor memory helper functions
  vhost: add mem region add/remove handlers
  vhost: enable configure memory slots

 .mailmap               |   1 +
 lib/vhost/rte_vhost.h  |   4 +
 lib/vhost/vhost_user.c | 438 +++++++++++++++++++++++++++++++++++------
 lib/vhost/vhost_user.h |  10 +
 4 files changed, 391 insertions(+), 62 deletions(-)

-- 
2.43.0