Intel QAT 8970 accel card on ARM Ampere Server

[Patrick Robb]

Sorry about the wait on my reply guys.

Thanks for the information. So I download the 2 diffs from that thread,
make a patch with them. Then where and how do I apply it?

Then I install the packages needed per the ubuntu page, and then I can skip
down to the "Building The Kernel" section? And then we're all set I think,
and we just have to setup DTS and associated Jenkins pipelines.

Do you want me to back anything up in advance of this? I don't know if that
is needed or not, but Ampere is currently live doing testing for CI, so I
want to act in a safe manner. I will try to address this first thing on
Monday and get back to you.



On Tue, Aug 8, 2023 at 3:11 AM Ruifeng Wang <Ruifeng.Wang at arm.com> wrote:

> *From:* Juraj Linkeš <juraj.linkes at pantheon.tech>
> *Sent:* Tuesday, August 8, 2023 3:07 PM
> *To:* Ruifeng Wang <Ruifeng.Wang at arm.com>
> *Cc:* Patrick Robb <probb at iol.unh.edu>; Honnappa Nagarahalli <
> Honnappa.Nagarahalli at arm.com>; Dharmik Jayesh Thakkar <
> DharmikJayesh.Thakkar at arm.com>; ci at dpdk.org; nd <nd at arm.com>
> *Subject:* Re: Intel QAT 8970 accel card on ARM Ampere Server
>
>
>
> We've talked about this some more and the best way to move forward is to
> rebuild the ubuntu kernel. It should be fairly straightforward according to their
> wiki page <https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel>. The page
> mentions a fairly old release (19.04), but was updated a year ago so the
> instructions are likely still valid.
>
>
>
> However, I don't have the link to the kernel patch that Honnappa
> mentioned. @Honnappa Nagarahalli <Honnappa.Nagarahalli at arm.com> @Ruifeng
> Wang <Ruifeng.Wang at arm.com>, can you please provide a reference for the
> patch?
>
> *[Ruifeng]* Here is the kernel patch set:
> https://lkml.org/lkml/2022/6/17/328
>
>
>
> Since the patch is small, there shouldn't be problems with applying it.
> Let us know whether this is doable.
>
>
>
> Regards,
>
> Juraj
>
>
>
> On Fri, Aug 4, 2023 at 11:48 AM Ruifeng Wang <Ruifeng.Wang at arm.com> wrote:
>
> Hi Patrick,
>
>
>
> Thanks for reaching out and my apologies for delayed response.
>
>
>
> We noticed that some information is missing regarding using QAT with DPDK
> on Arm.
>
> The DPDK document will be updated to include the missing part.
>
> Will get back on this later.
>
>
>
> Best regards,
>
> Ruifeng
>
>
>
> *From:* Patrick Robb <probb at iol.unh.edu>
> *Sent:* Tuesday, August 1, 2023 1:14 AM
> *To:* Ruifeng Wang <Ruifeng.Wang at arm.com>; Honnappa Nagarahalli <
> Honnappa.Nagarahalli at arm.com>; Juraj Linkeš <juraj.linkes at pantheon.tech>
> *Cc:* Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar at arm.com>; ci at dpdk.org
> *Subject:* Intel QAT 8970 accel card on ARM Ampere Server
>
>
>
> Hi Ruifeng, Honnappa, Juraj,
>
>
>
> The Intel QAT 8970 accelerator card has arrived to the Community Lab, and
> we've installed it on the Ampere server. Presumably, we should test both
> crypto and compress operations (and their respective performance metrics).
> To that end, there are also DTS testsuites for testing QAT crypto/compress
> functions. These testsuites make use of the crypto perf dpdk app and the
> compress perf dpdk app. If you want, you can setup the DTS stuff yourself,
> both on the system side, and the Jenkins side (you are allowed to submit
> PRs on our gitlab now), but we can also do that on the lab side as we
> probably have more experience. I do, however, have a question about the QAT
> kernel driver and corresponding PMDs.
>
>
>
> compress suite:
> https://git.dpdk.org/tools/dts/tree/test_plans/compressdev_qat_pmd_test_plan.rst
>
> crypto suite:
> https://git.dpdk.org/tools/dts/tree/test_plans/crypto_perf_cryptodev_perf_test_plan.rst
>
>
>
> For reference, the DPDK docs page explaining QAT driver capabilities and
> building the QAT PMDs (crypto sym, crypto asym, and compress) is here:
> https://doc.dpdk.org/guides/cryptodevs/qat.html#building-qat
>
> Some notes before I get to my main question:
>
> -The 8970 is a C62x device
>
> -OpenSSL (arm requires it for QAT) is installed
>
> -3 PFs are visible from lspci (expected)
>
> -SRIOV is enabled
>
>
>
> However, although the system is on a valid kernel version for the QAT
> driver, the kernel module for QAT is not loaded, so in trying to set up
> testing, I am unable to create the 16 VFs for the 3 PFs respectively, like
> the example below:
>
>
>
> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs
> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs
> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs
>
>
>
> There is also an option to download the firmware from the kernel firmware
> repo and copy the qat binaries to /lib/firmware and start the qat modules
> from there. I wasn't able to resolve the situation with this method, but it
> also could have been user error on my part.
>
>
>
> There is an option to install using the IDZ QAT Driver
> <https://www.intel.com/content/www/us/en/developer/topic-technology/open/quick-assist-technology/overview.html>,
> but it should not be required given the kernel version the Ampere server is
> on, and I don't want to go down the road of relying on this "fall back"
> method without consulting you first. Is it possible that there is anything
> specific to running a QAT device on ARM specifically which I am missing
> here? The DTS testsuite testplans actually seem to recommend going down
> this road in general, but the DPDK docs say to use the kernel driver, so I
> don't know.
>
>
>
> In any case, one of you should be able to login to the Ampere server in
> situations like this, or just in general. Ruifeng/Juraj I see you both have
> accounts on our IdM system, so you should have access. Please let me know
> if you need renewed vpn cert configs and I will send you one. If you do
> login, know this system could be running CI testing at any time. I can
> always schedule time for it to be offline and available for maintenance if
> you want to do anything which could be disruptive to testing.
>
>
>
> I also CC'd Dharmik on this as I see he sent an email regarding QAT
> support on aarch64 in June.
>
>
>
> Let me know if you have any thoughts on the QAT kernel driver part.
>
>
>
> Thanks,
>
> Patrick
>
>

-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mails.dpdk.org/archives/ci/attachments/20230811/edc3faab/attachment.htm>