[dpdk-dev] [PATCH v4 0/8] Windows basic memory management

[Dmitry Kozlyuk]

Note: symbols and release notes updated for v20.05, despite this patch
      now targeting v20.08, because proper sections don't yet exist.

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing with IOVA unavailable.

New EAL public functions for memory mapping are introduced
to mitigate OS differences in DPDK libraries and applications:

* rte_mem_map
* rte_mem_unmap
* rte_mem_lock
* rte_get_page_size

To support common MM routines, internal wrappers for low-level
memory reservation and file management are introduced. These changes
affect Linux and FreeBSD EAL. Shared code is placed unded /unix/
subdirectory (suggested by Thomas).

Also, entire <sys/queue.h> is imported from FreeBSD, replacing existing
partial import. There is already a license exception for this file.


Windows MM duplicates quite a lot of code from Linux EAL:

* eal_memalloc_alloc_seg_bulk
* eal_memalloc_free_seg_bulk
* calc_num_pages_per_socket
* rte_eal_hugepage_init

Perhaps this should be left as-is until Windows MM evolves into having
some specific requirements for these parts.


Notes on checkpatch warnings:

* No space after comma / no space before closing parent in macros---
  definitely a false-positive, unclear how to suppress this.
* Issues from imported BSD code---probably should be ignored?

---

v4:

    * Rebase on ToT, drop patches merged into master.
    * Rearrange patches to split Windows code (Jerin).
    * Fix Linux and FreeBSD build with make (Ophir).
    * Use int instead of enum to hold a set of flags (Anatoly).
    * Rename eal_mem_reserve items and fix their description (Anatoly).
    * Add eal_mem_set_dump() wrapper around madvise (Anatoly).
    * Don't claim Windows Server 2016 support due to lack of API (Tal).
    * Replace enum rte_page_sizes with a set of #defines (Jerin).
    * Fix documentation, SPDX tags, logging (Thomas).

v3:

    * Fix Linux build on and aarch64 and 32-bit x86 (reported by CI).
    * Fix logic and error handling while allocating segments.
    * Fix Unix rte_mem_map(): return NULL on failure.
    * Fix some checkpatch.sh issues:
        * Do not return positive errno, use DWORD for GetLastError().
        * Make dpdk-kmods source files non-executable.
    * Improve GSG for Windows Server (suggested by Ranjit Menon).

v2:

    * Rebase on ToT. Move all new code shared between Linux and FreeBSD
      to /unix/ subdirectory, also factor out some existing code there.
    * Improve description of Clang issue with rte_page_sizes on Windows.
      Restore -fstrict-enum for EAL. Check running, not target compiler.
    * Use EAL prefix for private facilities instead if RTE.
    * Improve documentation comments for new functions.
    * Remove co-installer for virt2phys. Add a typecast for clarity.
    * Document virt2phys in user guide, improve its own README.
    * Explicitly and forcefully disable multi-process.


Dmitry Kozlyuk (8):
  eal: replace rte_page_sizes with a set of constants
  eal: introduce internal wrappers for file operations
  eal: introduce memory management wrappers
  eal: extract common code for memseg list initialization
  eal/windows: replace sys/queue.h with a complete one from FreeBSD
  eal/windows: improve CPU and NUMA node detection
  eal/windows: initialize hugepage info
  eal/windows: implement basic memory management

 config/meson.build                            |   12 +-
 doc/guides/rel_notes/release_20_05.rst        |    2 +
 doc/guides/windows_gsg/build_dpdk.rst         |   20 -
 doc/guides/windows_gsg/index.rst              |    1 +
 doc/guides/windows_gsg/run_apps.rst           |   95 ++
 lib/librte_eal/common/eal_common_fbarray.c    |   56 +-
 lib/librte_eal/common/eal_common_memory.c     |  116 +-
 lib/librte_eal/common/eal_common_memzone.c    |    7 +
 lib/librte_eal/common/eal_private.h           |  155 ++-
 lib/librte_eal/common/meson.build             |   10 +
 lib/librte_eal/common/rte_malloc.c            |    9 +
 lib/librte_eal/freebsd/Makefile               |    5 +
 lib/librte_eal/freebsd/eal_memory.c           |   57 +-
 lib/librte_eal/include/rte_memory.h           |  109 +-
 lib/librte_eal/linux/Makefile                 |    5 +
 lib/librte_eal/linux/eal_memalloc.c           |    5 +-
 lib/librte_eal/linux/eal_memory.c             |   68 +-
 lib/librte_eal/meson.build                    |    4 +
 lib/librte_eal/rte_eal_exports.def            |  119 ++
 lib/librte_eal/rte_eal_version.map            |    4 +
 lib/librte_eal/unix/eal_unix.c                |   51 +
 lib/librte_eal/unix/eal_unix_memory.c         |  144 ++
 lib/librte_eal/unix/meson.build               |    7 +
 lib/librte_eal/windows/eal.c                  |  158 +++
 lib/librte_eal/windows/eal_hugepages.c        |  108 ++
 lib/librte_eal/windows/eal_lcore.c            |  185 ++-
 lib/librte_eal/windows/eal_memalloc.c         |  418 ++++++
 lib/librte_eal/windows/eal_memory.c           | 1155 +++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               |  103 ++
 lib/librte_eal/windows/eal_windows.h          |  100 ++
 lib/librte_eal/windows/include/meson.build    |    1 +
 lib/librte_eal/windows/include/rte_os.h       |    4 +
 .../windows/include/rte_virt2phys.h           |   34 +
 lib/librte_eal/windows/include/rte_windows.h  |    2 +
 lib/librte_eal/windows/include/sys/queue.h    |  663 +++++++++-
 lib/librte_eal/windows/include/unistd.h       |    3 +
 lib/librte_eal/windows/meson.build            |    6 +
 37 files changed, 3655 insertions(+), 346 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/unix/eal_unix.c
 create mode 100644 lib/librte_eal/unix/eal_unix_memory.c
 create mode 100644 lib/librte_eal/unix/meson.build
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h

-- 
2.25.1