[dpdk-dev] [PATCH v13 2/2] kni: support IOVA mode

[vattunuru at marvell.com]

From: Vamsi Attunuru <vattunuru at marvell.com>

Current KNI implementation only operates in IOVA_PA mode
patch adds required functionality to enable KNI in
IOVA_VA mode.

KNI loopback mode tests will have performance impact in
this mode due to IOVA to KVA address translations.
However, In KNI real world use cases, the performace
impact will be based on Linux kernel stack and scheduler
latencies. Performance varies based on the KNI use case.
If bus iommu scheme is IOVA_DC and KNI module is loaded,
DPDK chooses IOVA as PA as existing behaviour.

During KNI creation, app's iova_mode details are passed to
the KNI kernel module, accordingly kernel module translates
PA/IOVA addresses to KVA and vice-versa.

Signed-off-by: Vamsi Attunuru <vattunuru at marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark at marvell.com>
Suggested-by: Ferruh Yigit <ferruh.yigit at intel.com>
---
 doc/guides/prog_guide/kernel_nic_interface.rst | 15 +++++++++++++++
 doc/guides/rel_notes/release_19_11.rst         | 15 ++++++++++++++-
 lib/librte_eal/linux/eal/eal.c                 | 23 ++++++++++++++++-------
 lib/librte_kni/rte_kni.c                       |  6 ++++++
 4 files changed, 51 insertions(+), 8 deletions(-)

diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
index 2fd58e1..3bed18f 100644
--- a/doc/guides/prog_guide/kernel_nic_interface.rst
+++ b/doc/guides/prog_guide/kernel_nic_interface.rst
@@ -300,6 +300,21 @@ The sk_buff is then freed and the mbuf sent in the tx_q FIFO.
 The DPDK TX thread dequeues the mbuf and sends it to the PMD via ``rte_eth_tx_burst()``.
 It then puts the mbuf back in the cache.
 
+IOVA = VA: Support
+------------------
+
+KNI operates in IOVA_VA scheme when
+
+- LINUX_VERSION_CODE >= KERNEL_VERSION(4, 6, 0) and
+- eal option `iova-mode=va` is passed or bus IOVA scheme in the DPDK is selected
+  as RTE_IOVA_VA.
+
+KNI loopback mode tests will have performance impact in this mode due to IOVA to
+KVA address translations. However, In KNI real world use cases, the performace
+impact will be based on Linux kernel stack and scheduler latencies. Performance
+varies based on the KNI use case. If bus iommu scheme is IOVA_DC and KNI module
+is loaded, DPDK chooses IOVA as PA as existing behaviour.
+
 Ethtool
 -------
 
diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index 682c1bd..c207c93 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -292,8 +292,21 @@ New Features
   compilers store their internal representation of the source code that
   the linker uses at the final stage of compilation process.
 
-  See :doc:`../prog_guide/lto` for more information:
+* **Added IOVA as VA support for KNI.**
+
+  * Added IOVA = VA support for KNI, KNI can operate in IOVA = VA mode when
+    `iova-mode=va` eal option is passed to the application or when bus IOVA
+    scheme is selected as RTE_IOVA_VA. This mode only works on Linux Kernel
+    versions 4.6.0 and above.
 
+  * KNI loopback mode tests will have performance impact in this mode due
+    to IOVA to KVA address translations. However, In KNI real world use cases,
+    the performace impact will be based on Linux kernel stack and scheduler
+    latencies. Performance varies based on the KNI use case. If bus iommu
+    scheme is IOVA_DC and KNI module is loaded, DPDK chooses IOVA as PA as
+    existing behaviour.
+
+  See :doc:`../prog_guide/lto` for more information:
 
 
 Removed Items
diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
index 9e2d50c..53ca84b 100644
--- a/lib/librte_eal/linux/eal/eal.c
+++ b/lib/librte_eal/linux/eal/eal.c
@@ -1086,14 +1086,23 @@ rte_eal_init(int argc, char **argv)
 			}
 		}
 #ifdef RTE_LIBRTE_KNI
-		/* Workaround for KNI which requires physical address to work */
-		if (iova_mode == RTE_IOVA_VA &&
-				rte_eal_check_module("rte_kni") == 1) {
-			if (phys_addrs) {
+		if (rte_eal_check_module("rte_kni") == 1) {
+#if KERNEL_VERSION(4, 6, 0) > LINUX_VERSION_CODE
+			if (iova_mode == RTE_IOVA_VA) {
 				iova_mode = RTE_IOVA_PA;
-				RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because KNI module is loaded\n");
-			} else {
-				RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
+				RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because "
+						"Kernel version supports only 'PA' mode for KNI module\n");
+			}
+#endif
+			if (rte_bus_get_iommu_class() == RTE_IOVA_DC)
+				iova_mode = RTE_IOVA_PA;
+
+			if (iova_mode == RTE_IOVA_PA) {
+				if (phys_addrs && is_iommu_enabled())
+					RTE_LOG(WARNING, EAL, "Forced IOVA as 'PA' because KNI module is loaded\n");
+
+				if (!phys_addrs)
+					RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
 			}
 		}
 #endif
diff --git a/lib/librte_kni/rte_kni.c b/lib/librte_kni/rte_kni.c
index 7fbcf22..a609d2f 100644
--- a/lib/librte_kni/rte_kni.c
+++ b/lib/librte_kni/rte_kni.c
@@ -6,6 +6,8 @@
 #error "KNI is not supported"
 #endif
 
+#include <linux/version.h>
+
 #include <string.h>
 #include <fcntl.h>
 #include <unistd.h>
@@ -97,10 +99,12 @@ static volatile int kni_fd = -1;
 int
 rte_kni_init(unsigned int max_kni_ifaces __rte_unused)
 {
+#if KERNEL_VERSION(4, 6, 0) > LINUX_VERSION_CODE
 	if (rte_eal_iova_mode() != RTE_IOVA_PA) {
 		RTE_LOG(ERR, KNI, "KNI requires IOVA as PA\n");
 		return -1;
 	}
+#endif
 
 	/* Check FD and open */
 	if (kni_fd < 0) {
@@ -302,6 +306,8 @@ rte_kni_alloc(struct rte_mempool *pktmbuf_pool,
 	kni->group_id = conf->group_id;
 	kni->mbuf_size = conf->mbuf_size;
 
+	dev_info.iova_mode = (rte_eal_iova_mode() == RTE_IOVA_VA) ? 1 : 0;
+
 	ret = ioctl(kni_fd, RTE_KNI_IOCTL_CREATE, &dev_info);
 	if (ret < 0)
 		goto ioctl_fail;
-- 
2.8.4