[dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Ravi Kerur]

On Thu, Feb 8, 2018 at 3:20 AM, Burakov, Anatoly <anatoly.burakov at intel.com>
wrote:

> On 06-Feb-18 5:55 PM, Ravi Kerur wrote:
>
>>
>> Hi Anatoly,
>>
>> I am actually confused with the state of vIOMMU + DPDK. Can you please
>> help me clarify?
>>
>> I tested following DPDK versions
>>
>> (1) DPDK 17.11, exhibits the issue (IOMMU width as reported by RedHat and
>> solution is to prevent using the patch)
>> (2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to bind a
>> device in VM with VFIO driver and no DMAR error message on host
>> (3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to bind a
>> device in VM with VFIO driver and no DMAR error message on host
>>
>> Clearly issue seen in 17.11 without the patch you mentioned is a
>> regression or the issue was masked in earlier DPDK version?  I did not test
>> traffic with any DPDK version because I wanted to first get DMAR errors on
>> host gone.
>>
>> Our application 'v1' is integrated with DPDK 16.11 and 'v2' is integrated
>> with DPDK 17.05.01. In both 'v1' and 'v2' cases I don't see IOMMU width
>> error messages on VM, however, DMAR error messages are seen host. I am not
>> able to relate what causes DMAR error messages on host?
>>
>>
>> Thanks.
>>
>>
> Hi Ravi,
>
> vIOMMU support is out of our hands, really - we can only make use of
> hardware (or emulation of it) that is available. 39-bit wide address *can*
> work, you just have to be lucky and get PA addresses that would fit into 39
> bits (so under 512G limit), because we set up IOVA addresses to be 1:1 to
> physical addresses. We could, in principle, set up IOVA addresses to go
> from zero instead of them being a 1:1 mapping to physical addresses, but
> that would introduce need to translate addresses between IOVA and physical
> in some cases (e.g. KNI).
>
> I'm not aware of any changes between 16.11 and 17.11 (and indeed 18.02)
> that would make or break support for 39-bit wide PA addresses for IOMMU. It
> is possible that VF/PF drivers do something differently which results in
> DMAR errors showing up sooner rather than later, but as far as VFIO support
> itself is concerned, there were no major changes in those releases.
>
>
Hi Anatoly,

Thank you for your explanation. I would like to ask one more thing as I
need to get v-iommu+ dpdk working in VM. Can you please tell me what
determines 'Host Address Width", I know my question has nothing to do with
dpdk and this is a dpdk list, but if you have any information please share
it? I googled and found couple of ways to influence 'Host Address Width =
46' in guest as well (since dpdk + iommu works fine on host and DMAR on
host reports address width as 46).

(1) Qemu has CPU param 'host-phys-bits' boolean, when set to true copies it
from host
(2) Qemu has 'phys-bits'  integer, when set to '46' should influence guest

Using above options when instantiating a VM doesn't help, Guest VM still
ends up with 'Host address width = 39'.

(3) There is another Qemu option 'x-aw-bits' which is for VT-d which can be
set to '39' or '48'. This doesn't help either.

Thanks.

-- 
> Thanks,
> Anatoly
>