[dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)

Chinmaya Dwibedy ckdwibedy at gmail.com
Thu Jun 23 13:48:49 CEST 2016
Previous message: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)
Next message: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi All,

 On host, I installed dpdk-2.2.0 and then run
'cryptodev_qat_autotest’. I found this test case to be failed in
dequeue burst API ().Upon looking into the below system logs (#dmesg),
it seems, when this function is called, the QAT

hardware fails to write the crypto packet to the DMA buffer.  When I
passed intel_iommu=off command in boot line, it works. I think, when
we have no IOMMU, "physical" address space is accessed directly by
hardware, so code works.

The issue has been observed if the system runs with the kernel
parameter “intel_iommu=on”.



[root at stag48 ~]# dmesg

[  794.746134] dmar: DRHD: handling fault status reg 2

[  794.746141] dmar: DMAR:[DMA Read] Request device [83:01.0] fault addr
3398c000

DMAR:[fault reason 02] Present bit in context entry is clear

[  794.746158] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt:
Uncorrectable Push/Pull Misc Error

memory status: No errors occurred - Transaction Id 0x0 - Error type reserved

Bus Operation Type Push - Id 0x80000

[  794.746167] Reset needed for device: icp_dev0

[  794.746167] Auto Reset disabled. Please reset device0

[root at stag48 ~]#



*Note: *The QAT VF (*83:01.0*) is bound to the DPDK UIO driver.



83:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function

        Subsystem: Intel Corporation Device 0000

        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0

        Region 0: [virtual] Memory at c8280000 (64-bit, non-prefetchable)
[size=4K]

        Region 2: [virtual] Memory at c82a0000 (64-bit, non-prefetchable)
[size=4K]

        Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+

                Address: 0000000000000000  Data: 0000

                Masking: 00000000  Pending: 00000000

        Kernel driver in use: igb_uio





Now testing the same on Guest and finding the same issue (i.e.,
failing on dequeue burst API ()). Upon looking into error message (in
dmesg) on host/hypervisor it says i.e., [28475.040313]

vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state. The
PCI device 88:04.6 is the QAT VF, being assigned to Guest. I am trying
to figure out what causes “buffer not found” issue .

Please suggest if anyone has an idea on it.



Here goes the output of dmesg



[27086.464014] vfio-pci 0000:88:04.6: enabling device (0000 -> 0002)

[27086.464040] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[27086.564610] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[27086.574052] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[27089.786623] kvm: zapping shadow pages for mmio generation wraparound

[27106.320042] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X

[28474.784373] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[28475.040313] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state

[28493.271950] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X





Regards,

Chinmaya







On Wed, Jun 22, 2016 at 1:39 PM, Chinmaya Dwibedy <ckdwibedy at gmail.com>
wrote:

> Hi Sergio,
>
>
> Thank your for clarification. I am using dpdk-2.2.0. Here goes our
> configuration on VM. Can you please have a look into the same and let me
> know if anything is wrong. Will try to run the autotest, ipsec-segw on host
> as per your suggestion.
>
>
> [root at vpn-s ~]# uname -r
>
> 3.12.9-301.fc20.x86_64
>
> [root at vpn-s ~]#
>
>
> [root at vpn-s ~]# cat /etc/redhat-release
>
> Fedora release 20 (Heisenbug)
>
> [root at vpn-s ~]#
>
>
>
>
> [root at vpn-s ~]# cat /proc/cmdline
>
> BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64
> root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8
> iommu=pt intel_iommu=on
>
> [root at vpn-s ~]#
>
>
> [root at vpn-s QAT]# grep -i "iommu.*enabled" /var/log/messages
>
> Jun 22 07:31:17 vpn-s kernel: [    0.000000] Intel-IOMMU: enabled
>
>
> [root at vpn-s QAT]#
>
> [root at vpn-s dpdk-2.2.0]# lspci -nn | grep 0443
>
> 00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443]
>
> [root at vpn-s dpdk-2.2.0]#
>
>
> Installed Intel qat driver installation on VM (downloaded from
> https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches)
> and followed the
>
> https://01.org/sites/default/files/page/330750-005_qat_gsg.pdf guide.
> Note, used the following installation options i.e., ./installer.sh install
> QAT1.6 guest. After this , it detects QAT VF.
>
>
> [root at vpn-s QAT]# service qat_service status
>
> There is 1 acceleration device(s) in the system:
>
>  icp_dev0 - type=dh895xcc, inst_id=0, node_id=0,  bdf=00:06:0, #accel=1,
> #engines=1, state=up
>
> [root at vpn-s QAT]#
>
>
> [root at vpn-s QAT]# lsmod | grep qa
>
> icp_qa_al_vf          813442  1
>
> [root at vpn-s QAT]#
>
>
>
> The below are DPDK configuration options in
> config/defconfig_x86_64-native-linuxapp-gcc.
>
>
> include "common_linuxapp"
>
>
>
> CONFIG_RTE_MACHINE="native"
>
> CONFIG_RTE_ARCH="x86_64"
>
> CONFIG_RTE_ARCH_X86_64=y
>
> CONFIG_RTE_ARCH_64=y
>
> CONFIG_RTE_TOOLCHAIN="gcc"
>
> CONFIG_RTE_TOOLCHAIN_GCC=y
>
> CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y
>
> CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=y
>
> CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
>
> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y
>
> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y
>
> CONFIG_RTE_LIBRTE_PMD_QAT=y
>
> CONFIG_RTE_QAT_PMD_MAX_NB_SESSIONS=128
>
> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y
>
> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y
>
> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y
>
> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y
>
>
>
> The below is my setup.sh
>
>
> export RTE_SDK=/opt/dpdk-2.2.0
>
> export RTE_TARGET=x86_64-native-linuxapp-gcc
>
> export RTE_ARCH=x86_64
>
> export AESNI_MULTI_BUFFER_LIB_PATH=/opt/code
>
> make config T=x86_64-native-linuxapp-gcc
>
> sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config
>
> make install T=x86_64-native-linuxapp-gcc
>
> mkdir -p /mnt/huge
>
> mount -t hugetlbfs nodev /mnt/huge
>
> echo 2048 >
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>
> sudo modprobe uio_pci_generic
>
> sudo modprobe uio
>
> rmmod igb_uio
>
> insmod
> ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko
>
> echo -n "0000:00:06.0" > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind
>
> echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id
>
> lspci -vvd:443
>
> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic eth1
>
> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic 00:03.0
>
> ./tools/dpdk_nic_bind.py -s
>
>
> Also modified the cryptodevs_init() ( examples/ipsec-secgw/ipsec-secgw.c)
> to rte_cryptodev_start(cdev_id); after rte_cryptodev_queue_pair_setup().
> The purpose of calling this to start the transmit and the receive units
> of the QAT device. It returns success. But still getting the same issue
> i.e. rte_cryptodev_dequeue_burst () does not return crypto packet. Note
> that, when I call rte_cryptodev_queue_pair_setup () with 3rd argument as
> NULL (in which case default configuration will be used) , it crashes. Is
> it a known issue?
>
>
>
>
>
> Regards,
>
> Chinmaya
>
> On Tue, Jun 21, 2016 at 9:10 PM, Sergio Gonzalez Monroy <
> sergio.gonzalez.monroy at intel.com> wrote:
>
>> On 21/06/2016 08:26, Chinmaya Dwibedy wrote:
>>
>>>
>>> Hi Sergio,
>>>
>>>
>>> As suggested by you, run ‘app/test' application and then run
>>> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in
>>> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request ()
>>> function (app/test/test_cryptodev.c).
>>>
>>> Debugged the qat_crypto_pkt_rx_burst() (in
>>> drivers/crypto/qat/qat_crypto.c) and found that , the value of resp_msg is
>>> 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not getting crypto
>>> packet (i.e., rte_mbuf) from the RX queue.  But I find that, in
>>> qat_crypto_pkt_tx_burst(), the  qat_alg_write_mbuf_entry() is being called.
>>> The qat_alg_write_mbuf_entry () prints qat_req  using rte_hexdump () and
>>> returns zero.
>>>
>>>
>>> One question, the base_addr + tail of tx_q (in
>>> qat_crypto_pkt_tx_burst()) should be same as base_addr + head (in
>>> qat_crypto_pkt_rx_burst()). Right ? Please feel free to correct me if I am
>>> wrong.
>>>
>>>
>>> Also can you please provide some pointers for further debugging? Would
>>> it be configuration issue? Note that, I am using VF in VM.
>>>
>>>
>> In theory  you should be able to run ipsec-secgw sample app with QAT VF
>> as long as the setup is right.
>> That the autotest fails would likely point to some possible issues with
>> the setup.
>>
>> Have you tried to run the autotest in the host?
>>
>> Sergio
>>
>>
>>> Thank you once again for your prompt response and great support.
>>>
>>>
>>> Regards,
>>>
>>> Chinmaya
>>>
>>>
>>> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy <
>>> sergio.gonzalez.monroy at intel.com <mailto:
>>> sergio.gonzalez.monroy at intel.com>> wrote:
>>>
>>>     On 20/06/2016 10:08, Chinmaya Dwibedy wrote:
>>>
>>>
>>>         Hi Sergio,
>>>
>>>
>>>         Agreed.  We might not dequeue the same amount of crypto ops we
>>>         just previously enqueued, it's asynchronous. But in this case,
>>>         I have sent just one UDP packet. So there will be one crypto
>>>         ops. Right?  Also I put a sleep (50) after the
>>>         rte_crypto_enqueue_burst() function in ipsec_processing()
>>>         (ipsec.c) , so as to allow more time  ( for QAT device) for
>>>         processing. Still getting the same result i.e., the
>>>         rte_crypto_dequeue_burst () functionreturns zero.
>>>
>>>
>>>         In case of S/W crypto device (i.e., AESNI),   the VM gets
>>>         inbound UDP packets on Port 1/eth1, encapsulates (after
>>>         consulting its SPD) in an IPsec ESP packet and sends to its
>>>         peer through Port 0/eth0 interface.
>>>
>>>
>>>         Yes, the security policy, security association and Routing
>>>         entries/configurations are exactly same. Please feel free to
>>>         let me know if you need additional information.
>>>
>>>
>>>     Could you try to run 'app/test' application then run
>>>     'cryptodev_qat_autotest' ? That is a functional test for cryptodev
>>>     QAT PMD.
>>>
>>>     Sergio
>>>
>>>
>>>
>>
>
Previous message: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)
Next message: [dpdk-users] Issue with ipsec-secgw sample application on VM using Intel QAT device (pass-through mode)
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the users mailing list