[PATCH] vfio: do not coalesce DMA mappings

[Gupta, Nipun]

Hi Xuan,

On 7/4/2023 1:36 PM, Ding, Xuan wrote:
> Hi Nipun,
> 
>> -----Original Message-----
>> From: Gupta, Nipun <Nipun.Gupta at amd.com>
>> Sent: Tuesday, July 4, 2023 2:54 PM
>> To: Ding, Xuan <xuan.ding at intel.com>; dev at dpdk.org; thomas at monjalon.net;
>> Burakov, Anatoly <anatoly.burakov at intel.com>; Yigit, Ferruh
>> <Ferruh.Yigit at amd.com>; David Marchand <david.marchand at redhat.com>
>> Cc: Agarwal, Nikhil <nikhil.agarwal at amd.com>; He, Xingguang
>> <xingguang.he at intel.com>; Ling, WeiX <weix.ling at intel.com>
>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
>>
>> Hi Xuan,
>>
>> Please see inline.
>>
>>> -----Original Message-----
>>> From: Ding, Xuan <xuan.ding at intel.com>
>>> Sent: Tuesday, July 4, 2023 10:43 AM
>>> To: Gupta, Nipun <Nipun.Gupta at amd.com>; dev at dpdk.org;
>>> thomas at monjalon.net; Burakov, Anatoly <anatoly.burakov at intel.com>;
>>> Yigit, Ferruh <Ferruh.Yigit at amd.com>
>>> Cc: Agarwal, Nikhil <nikhil.agarwal at amd.com>; He, Xingguang
>>> <xingguang.he at intel.com>; Ling, WeiX <weix.ling at intel.com>
>>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
>>>
>>> Hi Nipun,
>>>
>>>> -----Original Message-----
>>>> From: Ding, Xuan
>>>> Sent: Friday, June 30, 2023 1:58 PM
>>>> To: Nipun Gupta <nipun.gupta at amd.com>; dev at dpdk.org;
>>>> thomas at monjalon.net; Burakov, Anatoly <anatoly.burakov at intel.com>;
>>>> ferruh.yigit at amd.com
>>>> Cc: nikhil.agarwal at amd.com; He, Xingguang <xingguang.he at intel.com>;
>>>> Ling, WeiX <weix.ling at intel.com>
>>>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
>>>>
>>>> Hi Nipun,
>>>>
>>>> Replies are inline.
>>>>
>>>>> -----Original Message-----
>>>>> From: Nipun Gupta <nipun.gupta at amd.com>
>>>>> Sent: Friday, June 30, 2023 9:46 AM
>>>>> To: Ding, Xuan <xuan.ding at intel.com>; dev at dpdk.org;
>>>>> thomas at monjalon.net; Burakov, Anatoly <anatoly.burakov at intel.com>;
>>>>> ferruh.yigit at amd.com
>>>>> Cc: nikhil.agarwal at amd.com; He, Xingguang
>>>>> <xingguang.he at intel.com>; Ling, WeiX <weix.ling at intel.com>
>>>>> Subject: Re: [PATCH] vfio: do not coalesce DMA mappings
>>>>>
>>>>> Hi Xuan,
>>>>>
>>>>> Thanks for pointing out the issue and figuring out the patch which
>>>>> introduced this. If you have answers to below queries, please let me know:
>>>>>
>>>>> Is there any other test cases which tests "--no-huge" which pass?
>>>>
>>>> Yes, there are test cases adding "--no-huge" option to validate 4k
>>>> page size in async vhost.
>>>> Actually, the page size is decided by front-end, so I think this
>>>> case can be removed.
>>>>
>>>> Previously, testpmd can start with "--no-huge" options (not sure if
>>>> there are test cases).
>>>> Cmd: ./build/app/dpdk-testpmd -l 5-6 -n 4 --no-huge -m 1024 -- -i
>>>>
>>>>>
>>>>> Also, if we change the "-m" option to provide lower memory, does
>>>>> the test pass?
>>>>
>>>> "-m" option is also added and does not work.
>>>>
>>>>>
>>>>> When you mention too many pages exceed the capability of IOMMU,
>>>>> you are referring to HW capability to create multiple pages? Here
>>>>> it seems in case of 4K page size we need 256K pages which is limiting the
>> capacity?
>>>>
>>>> Yes, this is the result of my initial debugging.
>>>> The direct impact is that this kind of testpmd cases cannot start now.
>>>> If this is expected, I think we can close this defect and ignore the "--no-
>> huge"
>>>> option when start.
>>>
>>> Any insights? Should we just ignore the "--no-huge" option and close this
>> defect?
>>> Now we did this as a workaround. Seems no one uses the "--no-huge"
>>> option in testpmd now.
>>
>> VFIO supports dma_entry_limit as a module parameter, which has a default
>> value of U16_MAX i.e. 64K, most likely which is limiting creation of 256K
>> entries for 4K pages here. This can be modified while inserting vfio module:
>>          modprobe vfio_iommu_type1 dma_entry_limit=1000000
> 
> Thanks for your suggestion. I tried it on ubuntu 22.04 but it does not work.
> The reason I think is vfio-pci is build-in in kernel driver (since 20.04) and it does not support dynamic insmod/rmmod.
> 
> Does this command need to rmmod vfio first and then modprobe again?
> 

If it is inserted as a module then you can remove using rmmod and then 
modprobe again with the dma_entry_limit parameter. Also note, 
vfio_iommu_type1 is the module which is limiting the entries to 64K, so 
this module needs to be inserted again providing the dma_entry_limit 
module param.

In case the module is built-in you can provide via kernel command line 
parameter (ref: 
https://www.kernel.org/doc/html/v4.12/admin-guide/kernel-parameters.html). 
As per this ref document, "vfio_iommu_type1.dma_entry_limit=1000000" 
should be used in the bootargs to set the module parameters.

FYI.. DPDK documentation also mentions the limitation at:
https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html

Thanks,
Nipun