[dpdk-users] Hard Crash with X710 and Pktgen

[Wiles, Keith]

Sent from my iPhone

On Nov 14, 2017, at 2:59 AM, Johnson, Brian <brian.johnson at intel.com<mailto:brian.johnson at intel.com>> wrote:

Do you have Intel VT-d enabled in the BIOS and intel_iommu=on iommu=pt in grub?

I have had some issues with enabling this where other devices such as RAID controllers cause the OS to no boot after applying these settings. Removing the grub line and upgrading the RAID firmware fixed the issue.

IOMMU<https://dpdk-guide.gitlab.io/dpdk-guide/setup/iommu.html#iommu>

An input-output memory management unit (IOMMU) is required for safely driving DMA-capable hardware from userspace and because of that it is a prerequisite for using VFIO. Not all systems have one though, so you’ll need to check that the hardware supports it and that it is enabled in the BIOS settings (VT-d or Virtualization Technology for Directed I/Oon Intel systems)

Finally, IOMMU needs to be excplitly enabled in the kernel as well. To do so, pass either intel_iommu=on (for Intel systems) or amd_iommu=on (for AMD systems) added to the kernel command line. In addition it is recommended to use iommu=pt option which improves IO performance for devices in the host.

Once the system boots up, check the contents of /sys/kernel/iommu_groups/ directory. If it is non-empty, you have successfully set up IOMMU.

To permanently add this to the kernel commandline, append it to GRUB_CMDLINE_LINUX in /etc/default/grub and then execute:

# grub2-mkconfig -o /boot/grub2/grub.cfg

Sent from my iPhone

On Nov 14, 2017, at 12:38 AM, James Bensley <jwbensley at gmail.com<mailto:jwbensley at gmail.com>> wrote:

Hi guys,

Thank you all for your responses!


I initially thought this was unlikely to be a Pktgen specific issue
however I thought it worth mentioning I am using Pktgen just in case.

On 14 November 2017 at 04:49, Muhammad Zain-ul-Abideen
<zain2294 at gmail.com<mailto:zain2294 at gmail.com>> wrote:
where is -p argument

Do you mean upper case "-P"? Lower case “p” isn’t an option in “Pktgen
–-help” output

Pktgen once used -p Portman value and is not used anymore I still see that option being used in some command lines.



On 13 November 2017 at 19:35, James Bensley <jwbensley at gmail.com<mailto:jwbensley at gmail.com>> wrote:
$ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0 -w 09:00.1 -v -- -P -m [3-4].0 [5-6].1


On 13 November 2017 at 21:49, Wiles, Keith <keith.wiles at intel.com<mailto:keith.wiles at intel.com>> wrote:
For the hard lockup problem try using testpmd application and see if that the same problem if not then it will be next week before I can look at.

Let me know if testpmd works or not.

Thanks for taking the time to reply whilst traveling Keith!

Sadly I’m getting the same behaviour with testpmd:

$ ./dpdk-devbind.py --status-dev net

Network devices using DPDK-compatible driver
============================================
0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=
0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb
unused=igb_uio *Active*
0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb
unused=igb_uio
0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0
drv=tg3 unused=igb_uio
0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1
drv=tg3 unused=igb_uio
0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2
drv=tg3 unused=igb_uio *Active*
0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3
drv=tg3 unused=igb_uio

$ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2
EAL: Detected 16 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:02:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:02:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:09:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e

<< HARD LOCK-UP >>


$ ./dpdk-devbind.py --status-dev net

Network devices using DPDK-compatible driver
============================================
0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
drv=vfio-pci unused=igb_uio
0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
drv=vfio-pci unused=igb_uio

Network devices using kernel driver
===================================
0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb
unused=igb_uio,vfio-pci *Active*
0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb
unused=igb_uio,vfio-pci
0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0
drv=tg3 unused=igb_uio,vfio-pci
0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1
drv=tg3 unused=igb_uio,vfio-pci
0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2
drv=tg3 unused=igb_uio,vfio-pci *Active*
0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3
drv=tg3 unused=igb_uio,vfio-pci

$ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2
EAL: Detected 16 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:02:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:09:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL:   0000:09:00.0 failed to select IOMMU type
EAL: Requested device 0000:09:00.0 cannot be used
EAL: PCI device 0000:09:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL:   0000:09:00.1 failed to select IOMMU type
EAL: Requested device 0000:09:00.1 cannot be used
EAL: No probed ethernet devices
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
size=2176, socket=0
Done


On 14 November 2017 at 06:29, Rosen, Rami <rami.rosen at intel.com<mailto:rami.rosen at intel.com>> wrote:
Second, I think the root cause for not finding the ports is around this message ("failed to select IOMMU type") in:
...
EAL:   probe driver: 8086:1572 net_i40e
EAL:   0000:09:00.0 failed to select IOMMU type
EAL: Requested device 0000:09:00.0 cannot be used
EAL: PCI device 0000:09:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL:   0000:09:00.1 failed to select IOMMU type
EAL: Requested device 0000:09:00.1 cannot be used
...

What does " find /sys/kernel/iommu_groups/ -type l"
give ?

Could it be that 0000:09:00.0 and 0000:09:00.1 belongs to an IOMMU group in which there are other devices ?

To me (a layman) this looks correct, do you agree?

$ find /sys/kernel/iommu_groups/ -type l | grep 09
/sys/kernel/iommu_groups/36/devices/0000:09:00.0
/sys/kernel/iommu_groups/37/devices/0000:09:00.1

$ find /sys/kernel/iommu_groups/ -type l | grep -E "36|37"
/sys/kernel/iommu_groups/36/devices/0000:09:00.0
/sys/kernel/iommu_groups/37/devices/0000:09:00.1

Cheers,
James.