[dpdk-dev] about IOVA and namespace in DPDK 19.05

[Burakov, Anatoly]

On 02-May-19 12:31 AM, Robert Nie wrote:

> After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops?
> 

That depends.

The IOVA addresses are what you would colloquially refer to as 
"physical" addresses (in other words, the addresses that hardware uses 
to do DMA requests). The reason we call them IOVA addresses and not 
physical addresses is because these DMA addresses do not necessarily 
correspond to actual physical addresses, but can instead be remapped by 
platform's IOMMU to be arbitrary. The term "IOVA" covers both cases.

When IOVA mode is set to "IOVA as PA", real physical addresses are used 
for DMA requests, while when IOVA mode is set to "IOVA as VA", then 
virtual addresses are used instead. The issue you were seeing with IOVA 
as PA mode is most likely caused by the fact that whatever it was that 
you were allocating, required IOVA-contiguous memory.

Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow 
and shrink memory usage on the fly), but that change came with a 
trade-off - IOVA contiguousness is no longer guaranteed, because we have 
no control over which page addresses the kernel will give us. Using IOVA 
as VA mode allows to sidestep this limitations, because we are no longer 
"given" any IOVA addresses - we set them to be whatever we want.

If your driver does not depend on *real physical address* 
contiguousness, then IOVA as VA mode is perfectly safe to use. For 
example, if your driver is entirely software-based, or if you're relying 
on some kind of kernel infrastructure (such as what would be the case 
with PCAP or AF_PACKET driver) to do actual DMA requests, then using 
IOVA as VA mode is OK. If, on the other hand, your use case requires 
actual physical addresses to be contiguous (for example, if you are 
using KNI), it is not safe.

I am only passably familiar with our AF_XDP driver, but i'm assuming 
that since it's technically a software driver, it should work with IOVA 
as VA mode just fine.

However, if you run into problems, you may try running DPDK in legacy 
mode (using --legacy-mem EAL switch) - this will make it so that DPDK is 
initialized similarly to how old versions of DPDK worked, and will 
reserve big amounts of IOVA-contiguous memory at startup (at a cost of 
losing the ability to grow and shrink memory usage at runtime).

-- 
Thanks,
Anatoly