[dpdk-dev] [PATCH] vfio: try physical address if virtual address fails

[Zhang, Qi Z]

Hi William:

> -----Original Message-----
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Chas Williams
> Sent: Saturday, November 25, 2017 7:57 AM
> To: dev at dpdk.org
> Cc: skhare at vmware.com; Chas Williams <3chas3 at gmail.com>; Chas
> Williams <chas3 at att.com>
> Subject: [dpdk-dev] [PATCH] vfio: try physical address if virtual address fails
> 
> Some machines appear to have buggy DMAR mappings.  A typical mapping
> error looks like:
> 
>     DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> mapped address (7fc4fa800000)
>     DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> mapped address (7fc4fa800000)
>     DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> mapped address (7fc4fa800000)
>     DMAR: intel_iommu_map: iommu width (39) is not sufficient for the
> mapped address (7fc4fa800000)
> 
I met the same issue on some intel atom platform, the root cause is IOMMU only support 39 bit virtual address.
Not sure retry with physical address will be the right fix. I saw rte_eal_iova_mode is called at other place, it still take the virtual address as the mapped result, does that break something? 
So far the workaround may works by using --virtbase-addr to assign a address in range explicitly (for example 0x70,0000,0000)
Regards
Qi

> To work around this, attempt to do a physical address mapping if the virtual
> address mapping fails.
> 
> Fixes: e85a919286d2 ("vfio: honor IOVA mode before mapping")
> 
> Signed-off-by: Chas Williams <chas3 at att.com>
> ---
>  lib/librte_eal/linuxapp/eal/eal_vfio.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c
> b/lib/librte_eal/linuxapp/eal/eal_vfio.c
> index 58f0123..6250676 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
> @@ -35,6 +35,7 @@
>  #include <fcntl.h>
>  #include <unistd.h>
>  #include <sys/ioctl.h>
> +#include <stdbool.h>
> 
>  #include <rte_log.h>
>  #include <rte_memory.h>
> @@ -702,6 +703,7 @@ vfio_type1_dma_map(int vfio_container_fd)
>  	/* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */
>  	for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>  		struct vfio_iommu_type1_dma_map dma_map;
> +		int retried = false;
> 
>  		if (ms[i].addr == NULL)
>  			break;
> @@ -716,9 +718,15 @@ vfio_type1_dma_map(int vfio_container_fd)
>  			dma_map.iova = ms[i].iova;
>  		dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
> VFIO_DMA_MAP_FLAG_WRITE;
> 
> +retry:
>  		ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA,
> &dma_map);
> 
>  		if (ret) {
> +			if (!retried && rte_eal_iova_mode() == RTE_IOVA_VA) {
> +				dma_map.iova = ms[i].iova;


> +				retried = true;
> +				goto retry;
> +			}
>  			RTE_LOG(ERR, EAL, "  cannot set up DMA remapping, "
>  					  "error %i (%s)\n", errno,
>  					  strerror(errno));
> --
> 2.9.5
Regards
Qi