[dpdk-users] i40e with DPDK exits abruptly in rte_eal_init()

[Saurabh Mishra]

Hi Qian --

>Mishra, could you help answer below questions?

*>* What’s your i40e’s vmware driver on host?


We are doing PCI-passthrough of i40e.


vmnic7  0000:07:00.1 i40e        Up   10000Mbps Full   3c:fd:fe:04:11:c2
1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+

[root at localhost:~] vmkload_mod -s i40e |grep Version

 Version: Version 1.3.38, Build: 1331820, Interface: 9.2 Built on: Aug  5
2015

[root at localhost:~] vmkload_mod -s i40e

vmkload_mod module information

 input file: /usr/lib/vmware/vmkmod/i40e

 Version: Version 1.3.38, Build: 1331820, Interface: 9.2 Built on: Aug  5
2015

 License: GPL

 Required name-spaces:

  com.vmware.driverAPI#9.2.2.0

  com.vmware.vmkapi#v2_2_0_0

 Parameters:

  skb_mpool_max: int

    Maximum attainable private socket buffer memory pool size for the
driver.

  skb_mpool_initial: int

    Driver's minimum private socket buffer memory pool size.

  heap_max: int

    Maximum attainable heap size for the driver.

  heap_initial: int

    Initial heap size allocated for the driver.

  debug: int

    Debug level (0=none,...,16=all)

  RSS: array of int

    Number of Receive-Side Scaling Descriptor Queues: 0 = disable/default,
1-4 = enable (number of cpus)

  VMDQ: array of int

    Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable
(default =I40E_ESX_DEFAULT_VMDQ)

  max_vfs: array of int

    Number of Virtual Functions: 0 = disable (default), 1-128 = enable this
many VFs

[root at localhost:~]

>What’s the firmware of FVL? Is your FVL 4X10G or 2x10G?

Hmm..how do I check that? It has two ports.


>Could you tell us your detailed test step? I only launch one dpdk app, no
secondary process, but your case have 2 processes. When pass through the
PCI device, I just pass through 2 ports to the VM and start the VM in my
case.

So I was passing one port to VM. I do launch primary and a secondary
process.

This is the error we got. It seems we were not able to map to the VA. So I
moved the rte_eal_init() and mbuf alloc to a separate process and now it
seems to work with one primary and six secondary process.

EAL:   PCI memory mapped at 0x7ffff7faa000

EAL: Cannot mmap device resource file
/sys/bus/pci/devices/0000:1b:00.0/resource3 to address: 0x7ffff7fac000

EAL: Error - exiting with code: 1

  Cause: Requested device 0000:1b:00.0 cannot be used



Thanks,
/Saurabh


On Sun, Jan 24, 2016 at 5:17 PM, Xu, Qian Q <qian.q.xu at intel.com> wrote:

> We have tried the I40E driver with FVL4 on ESXi in r2.2 testing and PCI
> pass through can work but SRIOV will have issues. I didn’t see Mishra’s
> error . Now we are stuck at setting up the ESXi environment, and need
>
> Vmware’s support to set up the env again.
>
> Mishra, could you help answer below questions?
>
> 1.       What’s your i40e’s vmware driver on host?
>
> 2.       What’s the firmware of FVL? Is your FVL 4X10G or 2x10G?
>
> 3.       Could you tell us your detailed test step? I only launch one
> dpdk app, no secondary process, but your case have 2 processes. When pass
> through the PCI device, I just pass through 2 ports to the VM and start the
> VM in my case.
>
>
>
> Helin, rombar=0 seems for KVM only, on VMWARE, not see the issue.
>
>
>
>
>
> Thanks
>
> Qian
>
>
>
> *From:* Zhang, Helin
> *Sent:* Sunday, January 24, 2016 11:17 AM
> *To:* Saurabh Mishra; Thomas Monjalon; Xu, Qian Q
> *Cc:* users at dpdk.org; Mcnamara, John
> *Subject:* RE: [dpdk-users] i40e with DPDK exits abruptly in
> rte_eal_init()
>
>
>
> Hi Saurabh
>
>
>
> Were you talking about assigning NIC PF to the VMWARE guest?
>
> I remember that there is an issue of assigning PF to KVM guest, then
> rombar=0 is needed for grub.
>
> Thank you for the issue report!
>
>
>
> Can Qian tell the details of that?
>
>
>
> Regards,
>
> Helin
>
>
>
> *From:* Saurabh Mishra [mailto:saurabh.globe at gmail.com
> <saurabh.globe at gmail.com>]
> *Sent:* Saturday, January 23, 2016 1:30 AM
> *To:* Thomas Monjalon <thomas.monjalon at 6wind.com>
> *Cc:* users at dpdk.org; Zhang, Helin <helin.zhang at intel.com>; Mcnamara,
> John <john.mcnamara at intel.com>
> *Subject:* Re: [dpdk-users] i40e with DPDK exits abruptly in
> rte_eal_init()
>
>
>
> Hi,
>
>
>
> Thanks Thomas.. So this is the error we see on i40e PCI-passthrough of
> whole NIC:
>
>
>
>
>
> *Secondary Process:*
>
>
>
> EAL: Detected lcore 0 as core 0 on socket 0
>
> EAL: Detected lcore 1 as core 0 on socket 0
>
> EAL: Support maximum 128 logical core(s) by configuration.
>
> EAL: Detected 2 lcore(s)
>
> EAL: Setting up physically contiguous memory...
>
> EAL: Analysing 1024 files
>
> EAL: Mapped segment 0 of size 0x22800000
>
> EAL: Mapped segment 1 of size 0x200000
>
> EAL: Mapped segment 2 of size 0x200000
>
> EAL: Mapped segment 3 of size 0x57600000
>
> EAL: Mapped segment 4 of size 0x400000
>
> EAL: Mapped segment 5 of size 0x400000
>
> EAL: Mapped segment 6 of size 0x400000
>
> EAL: Mapped segment 7 of size 0x200000
>
> EAL: Mapped segment 8 of size 0x2200000
>
> EAL: Mapped segment 9 of size 0x200000
>
> EAL: Mapped segment 10 of size 0x800000
>
> EAL: Mapped segment 11 of size 0x600000
>
> EAL: Mapped segment 12 of size 0x800000
>
> EAL: Mapped segment 13 of size 0xa00000
>
> EAL: Mapped segment 14 of size 0x400000
>
> EAL: Mapped segment 15 of size 0x200000
>
> EAL: Mapped segment 16 of size 0x200000
>
> EAL: Mapped segment 17 of size 0x200000
>
> EAL: Mapped segment 18 of size 0x200000
>
> EAL: memzone_reserve_aligned_thread_unsafe(): memzone <RG_MP_log_history>
> already exists
>
> RING: Cannot reserve memory
>
> EAL: TSC frequency is ~1799997 KHz
>
> EAL: Master lcore 1 is ready (tid=f7fe78c0;cpuset=[1])
>
> EAL: PCI device 0000:03:00.0 on NUMA socket 0
>
> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>
> EAL:   Not managed by a supported kernel driver, skipped
>
> EAL: PCI device 0000:1b:00.0 on NUMA socket 0
>
> EAL:   probe driver: 8086:1572 rte_i40e_pmd
>
> EAL:   PCI memory mapped at 0x7fff6f1f3000
>
> EAL:   PCI memory mapped at 0x7ffff7faa000
>
> EAL: Cannot mmap device resource file
> /sys/bus/pci/devices/0000:1b:00.0/resource3 to address: 0x7ffff7fac000
>
> EAL: Error - exiting with code: 1
>
>   Cause: Requested device 0000:1b:00.0 cannot be used
>
>
>
>
>
> # ./dpdk-2.2.0/tools/dpdk_nic_bind.py --status
>
>
>
> Network devices using DPDK-compatible driver
>
> ============================================
>
> 0000:1b:00.0 'Device 1572' drv=uio_pci_generic unused=i40e
>
>
>
> Network devices using kernel driver
>
> ===================================
>
> 0000:03:00.0 'VMXNET3 Ethernet Controller' if=eth0 drv=vmxnet3
> unused=uio_pci_generic *Active*
>
>
>
> Other network devices
>
> =====================
>
> <none>
>
>
>
> # grep Huge /proc/meminfo
>
> AnonHugePages:    118784 kB
>
> HugePages_Total:    1024
>
> HugePages_Free:        0
>
> HugePages_Rsvd:        0
>
> HugePages_Surp:        0
>
> Hugepagesize:       2048 kB
>
> #
>
>
>
>
>
> *Primary Process:*
>
>
>
> EAL: Detected lcore 0 as core 0 on socket 0
>
> EAL: Detected lcore 1 as core 0 on socket 0
>
> EAL: Support maximum 128 logical core(s) by configuration.
>
> EAL: Detected 2 lcore(s)
>
> EAL: Setting up physically contiguous memory...
>
> EAL: cannot open /proc/self/numa_maps, consider that all memory is in
> socket_id 0
>
> EAL: Ask a virtual area of 0x22800000 bytes
>
> EAL: Virtual area found at 0x7fffd0a00000 (size = 0x22800000)
>
> EAL: Ask a virtual area of 0x200000 bytes
>
> EAL: Virtual area found at 0x7fffd0600000 (size = 0x200000)
>
> EAL: Ask a virtual area of 0x200000 bytes
>
> EAL: Virtual area found at 0x7fffd0200000 (size = 0x200000)
>
> EAL: Ask a virtual area of 0x57600000 bytes
>
> EAL: Virtual area found at 0x7fff78a00000 (size = 0x57600000)
>
> EAL: Ask a virtual area of 0x400000 bytes
>
> EAL: Virtual area found at 0x7fff78400000 (size = 0x400000)
>
> EAL: Ask a virtual area of 0x400000 bytes
>
> EAL: Virtual area found at 0x7fff77e00000 (size = 0x400000)
>
> EAL: Ask a virtual area of 0x400000 bytes
>
> EAL: Virtual area found at 0x7fff77800000 (size = 0x400000)
>
> EAL: Ask a virtual area of 0x200000 bytes
>
> EAL: Virtual area found at 0x7fff77400000 (size = 0x200000)
>
> EAL: Ask a virtual area of 0x2200000 bytes
>
> EAL: Virtual area found at 0x7fff75000000 (size = 0x2200000)
>
> EAL: Ask a virtual area of 0x200000 bytes
>
> EAL: Virtual area found at 0x7fff74c00000 (size = 0x200000)
>
> EAL: Ask a virtual area of 0x800000 bytes
>
> EAL: Virtual area found at 0x7fff74200000 (size = 0x800000)
>
> EAL: Ask a virtual area of 0x600000 bytes
>
> EAL: Virtual area found at 0x7fff73a00000 (size = 0x600000)
>
> EAL: Ask a virtual area of 0x800000 bytes
>
> EAL: Virtual area found at 0x7fff73000000 (size = 0x800000)
>
> EAL: Ask a virtual area of 0xa00000 bytes
>
> EAL: Virtual area found at 0x7fff72400000 (size = 0xa00000)
>
> EAL: Ask a virtual area of 0x400000 bytes
>
> EAL: Virtual area found at 0x7fff71e00000 (size = 0x400000)
>
> EAL: Ask a virtual area of 0x200000 bytes
>
> EAL: Virtual area found at 0x7fff71a00000 (size = 0x200000)
>
> EAL: Ask a virtual area of 0x200000 bytes
>
> EAL: Virtual area found at 0x7fff71600000 (size = 0x200000)
>
> EAL: Ask a virtual area of 0x200000 bytes
>
> EAL: Virtual area found at 0x7fff71200000 (size = 0x200000)
>
> EAL: Ask a virtual area of 0x200000 bytes
>
> EAL: Virtual area found at 0x7fff70e00000 (size = 0x200000)
>
> EAL: Requesting 1024 pages of size 2MB from socket 0
>
> EAL: TSC frequency is ~1799997 KHz
>
> EAL: Master lcore 1 is ready (tid=f7fe78a0;cpuset=[1])
>
> EAL: PCI device 0000:03:00.0 on NUMA socket 0
>
> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>
> EAL:   Not managed by a supported kernel driver, skipped
>
> EAL: PCI device 0000:1b:00.0 on NUMA socket 0
>
> EAL:   probe driver: 8086:1572 rte_i40e_pmd
>
> EAL:   PCI memory mapped at 0x7fff6f1f3000
>
> EAL:   PCI memory mapped at 0x7ffff7fac000
>
> #
>
>
>
> On Fri, Jan 22, 2016 at 2:12 AM, Thomas Monjalon <
> thomas.monjalon at 6wind.com> wrote:
>
> Hi,
>
> Thanks for asking.
> There are a couple of requests hidden in this message.
> The maintainer of i40e is Helin (CC'ed).
>
> 2016-01-21 14:54, Saurabh Mishra:
> > Hi,
> >
> > We have noticed that i40e if we do PCI-pass through of whole NIC to VM on
> > ESXi 6.0, the DPDK exits abruptly in rte_eal_init()?
>
> This looks to be a bug report :)
>
> > We are passing following parameters:
> >
> >     char *eal_argv[] = {"fakeelf",
> >
> >                         "-c2",
> >
> >                         "-n4",
> >
> >                         "--proc-type=primary",};
> >
> >
> > int ret = rte_eal_init(4, eal_argv);
> >
> > The code works with Intel '82599ES 10-Gigabit SFI/SFP+ ' adapter in
> > PCI-passthrough or SR-IOV mode however i40e it does not work.
>
> You probably have the feeling that i40e does not work as other PMDs,
> maybe wondering what are the special extended PCI configs.
>
> > [root at localhost:~] esxcfg-nics -l
> >
> > [.]
> >
> > vmnic6  0000:07:00.0 i40e        Up   10000Mbps Full   3c:fd:fe:04:11:c0
> > 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+
> >
> > vmnic7  0000:07:00.1 i40e        Up   10000Mbps Full   3c:fd:fe:04:11:c2
> > 1500   Intel Corporation Ethernet Controller X710 for 10GbE SFP+
> >
> >
> > We have turned on following config in DPDK:
> >
> > CONFIG_RTE_PCI_CONFIG=y
> >
> > CONFIG_RTE_PCI_EXTENDED_TAG="on"
> > CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC=y
> >
> >
> > Is there any special handling in DPDK for i40e adapter in terms of
> config?
>
> So you had no help when reading the code comments neither in the doc.
> Indeed the only doc about i40e is the SR-IOV VF page:
>         http://dpdk.org/doc/guides/nics/intel_vf.html
>
> Please Helin, check the issue and the lack of documentation.
> Thanks
>
>
>