[dpdk-dev] [RFC] eal/memory: introducing an option to set iova as va

[santosh]

Hi Bruce,


On Friday 02 June 2017 02:57 PM, Bruce Richardson wrote:
> On Fri, Jun 02, 2017 at 09:54:46AM +0530, santosh wrote:
>> Ping?
>>
>> On Wednesday 24 May 2017 09:41 PM, Santosh Shukla wrote:
>>
>>> Some NPU hardware like OCTEONTX follows push model to get
>>> the packet from the pktio device. Where packet allocation
>>> and freeing done by the HW. Since HW can operate only on
>>> IOVA with help of SMMU/IOMMU, When packet receives from the
>>> Ethernet device, It is the IOVA address(which is PA in existing scheme).
>>>
>>> Mapping IOVA as PA is expensive on those HW, where every
>>> packet needs to be converted to VA from PA/IOVA.
>>>
>>> This patch proposes the scheme where the user can set IOVA
>>> as VA by using an eal command line argument. That helps to
>>> avoid costly lookup for VA in SW by leveraging the SMMU
>>> translation feature.
>>>
>>> Signed-off-by: Santosh Shukla <santosh.shukla at caviumnetworks.com>
>>> ---
> Hi,
>
> I agree this is a problem that needs to be solved, but this doesn't look
> like a particularly future-proofed solution. Given that we should
> use the IOMMU on as many platforms as possible for protection, we
> probably need to find an automatic way for DPDK to use IO addresses
> correctly. Is this therefore better done as part of the VFIO and
> UIO-specific code in EAL - as that is the part that knows how the memory
> mapping is done, and in the VFIO case, what address ranges were
> programmed in. The mempool driver was something else I considered but it
> is probably too high a level to implement this.

The other approach which we evaluated, Its detail:
0) Introduce a new bus api whose job is to detect iommu capable devices on that
bus {/ are those devices bind to iommu capable driver or not?}. Let's call that
api rte_bus_chk_iommu_dev();

1) The scheme is like If _all_ the devices bind to iommu kdrv then return iova=va
2) Otherwise switch to default mode i.e.. iova=pa.
3) Based on rte_bus_chk_iommu_dev() return value, 
accordingly program iova=va Or iova=pa in vfio_type1/spapr_map(). 

4) User from the command line can always override iova=va, 
in case if he wants to default scheme( iova=pa mode). For that purpose - Introduce eal
option something like --iova-pa Or --override-iova Or --iova-default 
or some better name.

Proposed API snap:

enum iova_mode {
    iova_va;
    iova_pa;
    iova_unknown;
};

/**
 * Look for iommu devices on that Bus.
 * And find out that those devices bind to iommu
 * capable driver example vfio.
 *
 *
 * @return
 *      On success return valid iova mode (iova_va or iova_pa)
 *      On failure return iova_unkown.
 */
typedef int (*rte_bus_chk_iommu_dev_t)(void);


By this approach, 
- We can automatically detect iova is va or pa
and then program accordingly. 
- Also, the user can always switch to default iova mode.
- Drivers like dpaa2 can use this API to detect iova mode then 
program dma_map accordingly. Currently they are doing in ifdef-way.

Comments? thoughts? Or if anyone has better proposal then, please
suggest.

> So, in short, I don't particularly like this solution, but I could live
> with it as a short-term option. Longer term though, I think we need a
> better way to support using IO addresses rather than physical addresses
> - I just don't know what that would look like or where it would sit/live.
>
> /Bruce

Thanks,.