<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 10, 2023 at 5:59 PM Patrick Robb <<a href="mailto:probb@iol.unh.edu">probb@iol.unh.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 9, 2023 at 11:56 PM Dharmik Jayesh Thakkar <<a href="mailto:DharmikJayesh.Thakkar@arm.com" target="_blank">DharmikJayesh.Thakkar@arm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div> <div lang="EN-US"> <div> <p class="MsoNormal">Hi Patrick,<u></u><u></u></p> <p class="MsoNormal"><u></u> <u></u></p> <p class="MsoNormal">Can you provide the grub settings? Is iommu.passthrough=1 included?</p></div></div></div></blockquote><div><br></div><div>Sure. I'm not sure if you just wanted the kernel cmdline options or the whole grub config, but I assume you just meant kernel cmdline. Let me know if you meant more.</div><div><br></div><div>GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=39-79 nohz_full=39-79 rcu_nocbs=39-79 processor.max_cstate=1 intel_pstate=disable console=ttyS0,115200 console=tty0"<br><br>But, iommu.passthrough=1 is not included, so I can add that if we need to. Do you know that this won't have any bad implications for the (intel, nvidia, broadcom) NICs which we test on this server?</div><div> </div></div></div></blockquote><div><br></div><div>Just a note here, Patrick. The iommu kernel and intel_pstate parameters aren't supported on arm, so you can remove those. And when iommu.passthrouh=1, IOMMU is bypassed and intel_iommu doesn't do anything (and maybe isn't supported on arm, but that's not clear from <a href="https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt">the docs</a>), so that can be removed as well.<br></div><div><br></div><div>From what I can tell, using iommu.passthrough=1 is the standard, so if there are any negative implications, we should investigate them, but there shouldn't be anything major.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div lang="EN-US"><div><p class="MsoNormal"><u></u><u></u></p> <p class="MsoNormal"><u></u> <u></u></p> <p class="MsoNormal">Also, is qat_c62xvf loaded as well?</p></div></div></div></blockquote><div>qat_c62xvf is built in to the kernel also. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div lang="EN-US"><div><p class="MsoNormal"> </p></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div lang="EN-US"><div><p class="MsoNormal"><u></u><u></u></p> <p class="MsoNormal"><u></u> <u></u></p> <p class="MsoNormal">Finally, a few guidelines on the vfio driver:<u></u><u></u></p> <p class="MsoNormal">At times, we need to configure the vfio driver.<u></u><u></u></p> <p class="MsoNormal">On kernel vers. 5.9+ we need to load the vfio-pci driver with the additional parameter <b>disable_denylist=1</b><u></u><u></u></p> <p class="MsoNormal">Unload the vfio-pci driver if it is already loaded so that we can reload it with the correct parameters :<br> <b>sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio</b><u></u><u></u></p> <p class="MsoNormal">If you can't unload the vfio driver because it's been built into the kernel, you'll have to find another way to change VFIO parameters, or to rebuild your kernel with VFIO_PCI set as a module. Failing to do that, you might encounter issues later on when you try to bind the VFs to VFIO.<br> <br> Load the vfio-pci driver and bind it to QAT VFs device ids:<br> <b>sudo modprobe vfio-pci disable_denylist=1 enable_sriov=1 vfio-pci.ids=8086:37c9</b><br> <br> Enable no-iommu-mode:<br> <b>echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode</b><u></u><u></u></p> <p class="MsoNormal"> /sys/module/vfio/parameter is missing ?<u></u><u></u></p> <p class="MsoNormal">If /sys/module/vfio/parameters does not exist, you might be missing the kernel module VFIO_NOIOMMU<u></u><u></u></p> <p class="MsoNormal"><u></u> <u></u></p> <p class="MsoNormal"><b>Automatically set VFIO params on boot<u></u><u></u></b></p> <p class="MsoNormal">It's possible to set these parameters automatically on boot by creating a <b>/etc/modprobe.d/vfio-pci.conf </b>file with the parameters :<br> <b>cat /etc/modprobe.d/vfio-pci.conf</b><br> <b>options vfio enable_unsafe_noiommu_mode=1</b><br> <b>options vfio-pci disable_denylist=1 enable_sriov=1 vfio-pci.ids=8086:37c9<u></u><u></u></b></p> <p class="MsoNormal"><b><u></u> <u></u></b></p> <p class="MsoNormal">We haven’t encountered this issue in the past, so just making sure the configuration is correct. I don’t think having the driver static/loadable should make a difference, I will try with building statically on my setup.<u></u><u></u></p> <p class="MsoNormal"><u></u> <u></u></p> <p class="MsoNormal">Thank you!<u></u><u></u></p> <p class="MsoNormal"><u></u><br></p></div></div> </div></blockquote></div><div>Okay, this should be fine. Like I said, we are also running tests on NICs on this server. So, in our Jenkinsfiles scripts for running the testing, I will add a preliminary step only for QAT tests which runs: <br><b>sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio</b></div><div><b>sudo modprobe vfio-pci disable_denylist=1 enable_sriov=1 vfio-pci.ids=8086:37c9</b></div><div><b>echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode</b></div><div>(then run QAT tests)</div><div><br></div><div>And if running on NICs, have a preliminary step which runs </div><div><b>sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio</b><br></div><div><b>sudo modprobe vfio</b></div><div><b><br></b></div><div>David does this also sound reasonable to you, per your comment about isolating this setting to QAT card testing?</div><div><br></div><div>Dharmik if this all sounds okay and you can confirm the iommu.passthrough change is fine, I will proceed. Thank you for providing the assistance. <br></div><div><br></div></div> </blockquote></div></div>