[dpdk-dev] [PATCH v2 00/10] eal: Windows basic memory management

[Dmitry Kozlyuk]

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing with IOVA unavailable.


The first commit introduces a new kernel-mode driver, virt2phys.
It translates user-mode virtual addresses into physical addresses.
On Windows community call 2020-04-01 it was decided this driver can be
used for now, later netUIO may pick up its code/interface or not.


New EAL public functions for memory mapping are introduced
to mitigate OS differences in DPDK libraries and applications:

* rte_mem_map
* rte_mem_unmap
* rte_mem_lock

To support common MM routines, internal wrappers for low-level
memory reservation and file management are introduced. These changes
affect Linux and FreeBSD EAL. Shared code is placed unded /unix/
subdirectory (suggested by Thomas).

Also, entire <sys/queue.h> is imported from FreeBSD, replacing existing
partial import. There is already a license exception for this file.


Windows MM duplicates quite a lot of code from Linux EAL:

* eal_memalloc_alloc_seg_bulk
* eal_memalloc_free_seg_bulk
* calc_num_pages_per_socket
* rte_eal_hugepage_init

Perhaps this should be left as-is until Windows MM evolves into having
some specific requirements for these parts.


Notes on checkpatch warnings:

* No space after comma / no space before closing parent in macros---
  definitely a false-positive, unclear how to suppress this.

* Issues from imported BSD code---probably should be ignored?

* Checkpatch is not run against dpdk-kmods (Windows drivers).

---

There were comments from Mellanox and Microsoft outside of this mailing list.
Still missing input from Anatoly Burakov.

v2:

    * Rebase on ToT. Move all new code shared between Linux and FreeBSD
      to /unix/ subdirectory, also factor out some existing code there.
    * Improve description of Clang issue with rte_page_sizes on Windows.
      Restore -fstrict-enum for EAL. Check running, not target compiler.
    * Use EAL prefix for private facilities instead if RTE.
    * Improve documentation comments for new functions.
    * Remove co-installer for virt2phys. Add a typecast for clarity.
    * Document virt2phys in user guide, improve its own README.
    * Explicitly and forcefully disable multi-process.

Dmitry Kozlyuk (9):
  eal/windows: do not expose private EAL facilities
  eal/windows: improve CPU and NUMA node detection
  eal/windows: initialize hugepage info
  eal: introduce internal wrappers for file operations
  eal: introduce memory management wrappers
  eal: extract common code for memseg list initialization
  eal/windows: fix rte_page_sizes with Clang on Windows
  eal/windows: replace sys/queue.h with a complete one from FreeBSD
  eal/windows: implement basic memory management

 config/meson.build                            |   12 +-
 doc/guides/windows_gsg/build_dpdk.rst         |   20 -
 doc/guides/windows_gsg/index.rst              |    1 +
 doc/guides/windows_gsg/run_apps.rst           |   77 ++
 lib/librte_eal/common/eal_common_fbarray.c    |   57 +-
 lib/librte_eal/common/eal_common_memory.c     |  104 +-
 lib/librte_eal/common/eal_private.h           |  134 +-
 lib/librte_eal/common/malloc_heap.c           |    1 +
 lib/librte_eal/common/meson.build             |    9 +
 lib/librte_eal/freebsd/eal_memory.c           |   55 +-
 lib/librte_eal/include/rte_memory.h           |   74 ++
 lib/librte_eal/linux/eal_memory.c             |   68 +-
 lib/librte_eal/meson.build                    |    5 +
 lib/librte_eal/rte_eal_exports.def            |  119 ++
 lib/librte_eal/rte_eal_version.map            |    4 +
 lib/librte_eal/unix/eal.c                     |   47 +
 lib/librte_eal/unix/eal_memory.c              |  112 ++
 lib/librte_eal/unix/meson.build               |    7 +
 lib/librte_eal/windows/eal.c                  |  160 +++
 lib/librte_eal/windows/eal_hugepages.c        |  108 ++
 lib/librte_eal/windows/eal_lcore.c            |  187 +--
 lib/librte_eal/windows/eal_memalloc.c         |  423 ++++++
 lib/librte_eal/windows/eal_memory.c           | 1133 +++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               |  103 ++
 lib/librte_eal/windows/eal_thread.c           |    1 +
 lib/librte_eal/windows/eal_windows.h          |  129 ++
 lib/librte_eal/windows/include/meson.build    |    2 +
 lib/librte_eal/windows/include/pthread.h      |    2 +
 lib/librte_eal/windows/include/rte_os.h       |   48 +-
 .../windows/include/rte_virt2phys.h           |   34 +
 lib/librte_eal/windows/include/rte_windows.h  |   43 +
 lib/librte_eal/windows/include/sys/queue.h    |  663 +++++++++-
 lib/librte_eal/windows/include/unistd.h       |    3 +
 lib/librte_eal/windows/meson.build            |    4 +
 34 files changed, 3599 insertions(+), 350 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/unix/eal.c
 create mode 100644 lib/librte_eal/unix/eal_memory.c
 create mode 100644 lib/librte_eal/unix/meson.build
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/eal_windows.h
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
 create mode 100644 lib/librte_eal/windows/include/rte_windows.h

-- 
2.25.1