<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Patrick,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Can you provide the grub settings? Is iommu.passthrough=1 included?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Also, is qat_c62xvf loaded as well?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Finally, a few guidelines on the vfio driver:<o:p></o:p></p>
<p class="MsoNormal">At times, we need to configure the vfio driver.<o:p></o:p></p>
<p class="MsoNormal">On kernel vers. 5.9+ we need to load the vfio-pci driver with the additional parameter <b>disable_denylist=1</b><o:p></o:p></p>
<p class="MsoNormal">Unload the vfio-pci driver if it is already loaded so that we can reload it with the correct parameters :<br>
<b>sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio</b><o:p></o:p></p>
<p class="MsoNormal">If you can't unload the vfio driver because it's been built into the kernel, you'll have to find another way to change VFIO parameters, or to rebuild your kernel with VFIO_PCI set as a module. Failing to do that, you might encounter issues
later on when you try to bind the VFs to VFIO.<br>
<br>
Load the vfio-pci driver and bind it to QAT VFs device ids:<br>
<b>sudo modprobe vfio-pci disable_denylist=1 enable_sriov=1 vfio-pci.ids=8086:37c9</b><br>
<br>
Enable no-iommu-mode:<br>
<b>echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode</b><o:p></o:p></p>
<p class="MsoNormal"> /sys/module/vfio/parameter is missing ?<o:p></o:p></p>
<p class="MsoNormal">If /sys/module/vfio/parameters does not exist, you might be missing the kernel module VFIO_NOIOMMU<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Automatically set VFIO params on boot<o:p></o:p></b></p>
<p class="MsoNormal">It's possible to set these parameters automatically on boot by creating a <b>/etc/modprobe.d/vfio-pci.conf </b>file with the parameters :<br>
<b>cat /etc/modprobe.d/vfio-pci.conf</b><br>
<b>options vfio enable_unsafe_noiommu_mode=1</b><br>
<b>options vfio-pci disable_denylist=1 enable_sriov=1 vfio-pci.ids=8086:37c9<o:p></o:p></b></p>
<p class="MsoNormal"><b><o:p> </o:p></b></p>
<p class="MsoNormal">We haven’t encountered this issue in the past, so just making sure the configuration is correct. I don’t think having the driver static/loadable should make a difference, I will try with building statically on my setup.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Patrick Robb <probb@iol.unh.edu> <br>
<b>Sent:</b> Monday, October 9, 2023 9:29 PM<br>
<b>To:</b> Ruifeng Wang <Ruifeng.Wang@arm.com><br>
<b>Cc:</b> Juraj Linkeš <juraj.linkes@pantheon.tech>; Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; ci@dpdk.org; nd <nd@arm.com><br>
<b>Subject:</b> Re: Intel QAT 8970 accel card on ARM Ampere Server<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Also I am just now thinking I probably should have provided dpdk-devbind.py output:<br>
<br>
probb@arm-ampere-dut:/tmp/dpdk/usertools$ dpdk-devbind.py --status<br>
Crypto devices using kernel driver<br>
==================================<br>
0000:03:00.0 'C62x Chipset QuickAssist Technology 37c8' drv=c6xx unused=vfio-pci <br>
0000:04:00.0 'C62x Chipset QuickAssist Technology 37c8' drv=c6xx unused=vfio-pci <br>
0000:05:00.0 'C62x Chipset QuickAssist Technology 37c8' drv=c6xx unused=vfio-pci <o:p></o:p></p>
</div>
</div>
</div>
</div>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose,
or store or copy the information in any medium. Thank you.
</body>
</html>