[dpdk-users] [dpdk][pktgen] - execute pktgen with mlx4 (40G) with some errors

[Wiles, Keith]

> On Aug 1, 2017, at 3:15 AM, Paul Tsvika <mozloverinweb at gmail.com> wrote:
> 
> Hi zheng,
> 
> Thanks for the feedback.
> 
> You are right. To get pktgen work with mlx4, it needs to recompile DPDK and
> pktgen as well.
> 
> I am able to run pktgen now.
> 
> By following the document from DPDK website for this NIC,
> I did and change the following variables
> 
> I set *MLX4_INLINE_RECV_SIZE=64* as the environment variable in bashrc.
> set l*og_num_mgm_entry_size=-7*
> 
> Disable pause frames:
> *ethtool -A ens2d1 rx off tx 0ff*
> 
> I did not change the NIC register ( *setpci* ) at this time.
> 
> 
> and then I run the commands below in order:
> 
> 
> *test$: modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib*
> *test$: /etc/init.d/openibd restart*
> *test:$: sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-6 -n 1
> --proc-type auto --file-prefix pg -b 03:00.0 -b 03:00.1 -b 05:00.0 -b
> 05:00.1 -- -P -m "[1-5.0]"*
> 
> | Ports 0-1 of 2   <Main Page>  Copyright (c) <2010-2017>, Intel Corporation
>  Flags:Port      :   P--------------:0
> Link State        :          <--Down-->     ----TotalRate----
> Pkts/s Max/Rx     :                 0/0                   0/0
>       Max/Tx     :       844160/732319         844160/732319
> MBits/s Rx/Tx     :               0/*492 *                0/*492*


Note the link state is down and it needs to be up to determine the link speed for Pktgen.

> Broadcast         :                   0
> Multicast         :                   0
>  64 Bytes        :                   0
>  65-127          :                   0
>  128-255         :                   0
>  256-511         :                   0
>  512-1023        :                   0
>  1024-1518       :                   0
> Runts/Jumbos      :                 0/0
> Errors Rx/Tx      :                 0/0
> Total Rx Pkts     :                   0
>      Tx Pkts     :            59289215
>      Rx MBs      :                   0
>      Tx MBs      :               39842
> ARP/ICMP Pkts     :                 0/0
> Tx Overrun        :                   0
> Cycles per Tx     :       14880952/3200
> Missed Rx         :                   0
> mcasts Rx         :                None
> No Mbuf Rx        :                   0
>                  :
> Pattern Type      :             abcd...
> Tx Count/% Rate   :       Forever /100%
> PktSize/Tx Burst  :           64 /   32
> Src/Dest Port     :         1234 / 5678
> Pkt Type:VLAN ID  :     IPv4 / TCP:0001
> Dst  IP Address   :         192.168.1.1
> Src  IP Address   :      192.168.0.1/24
> Src MAC Address   :   e4:1d:2d:02:44:81
> Powered by DPDK --------------------------
> 
> The transmitting rate should be around 40000 since it is 40G NIC.
> However, the rate is much lower than it and it is probably related to
> configurations.
> 
> I actually have two mother boards and each one has 40G NICs installed.
> My intention is to do the tx / rx experiment.
> My first step is to check tx rate can reach as close to 40000 as possible.
> 
> Anyone has related experience please advise.
> 
> 
> 
> P.T
> 
> 
> 2017-08-01 12:15 GMT+08:00 zhilong zheng <zhengzl0715 at gmail.com>:
> 
>> 
>> Hi,
>> 
>> After enabling the MLX4 flag and recompiling the DPDK source code, Do you
>> recompile the code of pktgen? I think you can recompile it by running make
>> clean and make.
>> 
>> Regards,
>> Zhilong
>> 
>>> 在 2017年8月1日，11:38，Paul Tsvika <mozloverinweb at gmail.com> 写道：
>>> 
>>> Hi Olga,
>>> 
>>> 2017-08-01 0:11 GMT+08:00 Olga Shern <olgas at mellanox.com>:
>>> 
>>>> Hi Paul,
>>>> 
>>>> Looks like your ConnectX-3 configured to be IB and not ETH
>>>> 
>>>>> pci at 0000:04:00.0  ib0     network        MT27500 Family [ConnectX-3]
>>>> 
>>>> Use connectx_port_config script.
>>>> Set all the ports to Ethernet, and follow the instructions on the
>> screen.
>>>> 
>>> 
>>> Thanks. I followed the script and configured  both ports to Ethernet.
>>> 
>>> ibstat info below:
>>> 
>>> CA 'mlx4_0'
>>>   CA type: MT4099
>>>   Number of ports: 2
>>>   Firmware version: 2.40.7000
>>>   Hardware version: 1
>>>   Node GUID: 0xe41d2d0300024480
>>>   System image GUID: 0xe41d2d0300024483
>>>   Port 1:
>>>       State: Down
>>>       Physical state: Disabled
>>>       Rate: 10
>>>       Base lid: 0
>>>       LMC: 0
>>>       SM lid: 0
>>>       Capability mask: 0x04010000
>>>       Port GUID: 0xe61d2dfffe024481
>>>       Link layer: Ethernet
>>>   Port 2:
>>>       State: Active
>>>       Physical state: LinkUp
>>>       Rate: 40
>>>       Base lid: 0
>>>       LMC: 0
>>>       SM lid: 0
>>>       Capability mask: 0x04010000
>>>       Port GUID: 0xe61d2dfffe024482
>>>       Link layer: Ethernet
>>> 
>>> I also change the default value from n to  CONFIG_RTE_LIBRTE_MLX4_PMD=y.
>>> Recompiled DPDK and it all worked.
>>> 
>>> And Below is my ifconfig info:
>>> 
>>> ens2      Link encap:Ethernet  HWaddr e4:1d:2d:02:44:81
>>>         UP BROADCAST MULTICAST  MTU:1500  Metric:1
>>>         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>>         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>>         collisions:0 txqueuelen:1000
>>>         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>>> 
>>> ens2d1    Link encap:Ethernet  HWaddr e4:1d:2d:02:44:82
>>>         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>>         RX packets:126 errors:0 dropped:0 overruns:0 frame:0
>>>         TX packets:220 errors:0 dropped:0 overruns:0 carrier:0
>>>         collisions:0 txqueuelen:1000
>>>         RX bytes:29494 (29.4 KB)  TX bytes:38063 (38.0 KB)
>>> 
>>> 
>>> Since I assume that I don't have to bind the device manually, I executed
>>> ./dpdk-devbind.py --status with the following info popped out:
>>> 
>>> Network devices using kernel driver
>>> ===================================
>>> 0000:03:00.0 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' if=eno3
>>> drv=ixgbe unused=
>>> 0000:03:00.1 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' if=eno4
>>> drv=ixgbe unused=
>>> 0000:04:00.0 'MT27500 Family [ConnectX-3] 1003' if=ens2,ens2d1
>>> drv=mlx4_core unused=
>>> 0000:05:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb
>> unused=
>>> 0000:05:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb
>> unused=
>>> *Active*
>>> 
>>> 
>>> With these configurations, I run commands in order but still had the same
>>> result:
>>> 
>>> $test: modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib
>>> $test: /etc/init.d/openibd restart
>>> $test: sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-6 -n 1
>> --proc-type
>>> auto --file-prefix pg -b 03:00.0 -b 03:00.1 -b 05:00.0 -b 05:00.1 -- -P
>> -m
>>> "[1-5.0]"
>>> 
>>> 
>>>>>> Packet Burst 32, RX Desc 512, TX Desc 1024, mbufs/port 8192, mbuf
>> cache
>>> 1024
>>> !PANIC!: *** Did not find any ports to use ***
>>> PANIC in pktgen_config_ports():
>>> *** Did not find any ports to use ***6:
>>> [./app/x86_64-native-linuxapp-gcc/pktgen(_start+0x29)
>>> [0x447239]]
>>> 5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)
>>> [0x7f0ec0931830]]
>>> 4: [./app/x86_64-native-linuxapp-gcc/pktgen(main+0x580) [0x444710]]
>>> 3: [./app/x86_64-native-linuxapp-gcc/pktgen(pktgen_config_ports+0x1614)
>>> [0x46e324]]
>>> 2: [./app/x86_64-native-linuxapp-gcc/pktgen(__rte_panic+0xc3)
>> [0x43ead4]]
>>> 1: [./app/x86_64-native-linuxapp-gcc/pktgen(rte_dump_stack+0x2b)
>> [0x4b7ecb]]
>>> Aborted (core dumped)
>>> 
>>> 
>>> 
>>> Is there anything I've done wrong here?
>>> 
>>> 
>>> Thanks
>>> 
>>> 
>>> P.T
>>> 
>>> 
>>>> You can use  -w to specify ConnectX-3 PCI address, you don't need to
>> use -b
>>>> 
>>>> Make sure you compile DPDK with mlx4 PMD, it is not enabled by default
>>>> 
>>>> Best Regards,
>>>> Olga
>>>> 
>>>> 
>>>> 
>>> 
>>> --
>>> P.T
>> 
> 
> 
> 
> -- 
> P.T

Regards,
Keith