[dpdk-dev] pktgen-dpdk send too big packet, and stop sending packets after few seconds

Wiles, Keith keith.wiles at intel.com
Sun Jun 23 15:26:15 CEST 2019



> On Jun 23, 2019, at 3:00 AM, Xia Rui <xiarui_work at 163.com> wrote:
> 
> Hello, everyone.
> I am using pktgen-dpdk and testpmd to test the functionality of ovs-dpdk. The network topology is :
> +-------------+----------------------+           host(OVS-DPDK)         +-----------------------+-----------------+
> |                  | vhost-user port 1 |<----------------------------------->| vhost-user port 3 |                       |
> |                  +----------------------+                                               +-----------------------+                     |
> | container   |         pktgen        |                                                 |         testpmd        |     container |
> |                  +----------------------+                                                +-------------------+                         |
> |                  | vhost-user port 2 |<------------------------------------>| vhost-user port 4 |                     |
> +--------------+---------------------+                                                 +----------------------+----------------+
> 
> The version of my platform:
> 1. host OS: ubuntu 16.04.5 LTS
> 2. host linux kernel: 4.15.0-15
> 3. host OVS: 2.8.0
> 4. host DPDK : 17.05.2
> 5. container pktgen-dpdk: 3.4.9 + DPDK 17.05.2
> 6. container DPDK (testpmd): 17.05.2

At one point virtio drivers were changing the length of the tx packet in the mbuf for some type of meta data being shared between the two virtio points. Pktgen expected the length to remain the same when the mbuf was returned to pktgen mempool. This explains the non-64 byte frames. The fix was added to virtio in later releases to restore the length in the mbuf, plus I may’ve added code to pktgen in later release to fix the length back to what I expected when sending the packets. I looked in the latests version of pktgen and found the code in pktgen_setup_cb() which reset the length in the packet. The code should call rte_pktmbuf_reset() routine to restore the mbuf to expected state.

If OVS can work with a later version of DPDK then I would upgrade to the latest release, if not then I would look at and compare the two versions of virtio PMD and see if you can find that fix. If not then you can look in pktgen and repair the length in the mbuf before it is sent plus you may have to fix the location of the read/write offsets in the mbuf as well. This change will effect pktgen performance, but for virtio that should not be a problem. You can also connect the Maintainers of virtio and see if they remember the fix.

For the traffic stopping, I do not remember if this fix solved that problem.
> 
> There are two docker containers. One is running pktgen-dpdk with:
> ############################pktgen-dpdk start script############################
> ./app/x86_64-native-linuxapp-gcc/pktgen -c 0x70 --master-lcore 4 -n 1 --file-prefix pktgen --no-pci \
> --vdev 'net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1' \
> --vdev 'net_virtio_user2,mac=00:00:00:00:00:02,path=/var/run/openvswitch/vhost-user2' \
> -- -T -P -m "5.0,6.1"
> ############################pktgen-dpdk start script END############################
> 
> The other is running testpmd with:
> ############################testpmd start script############################
> testpmd -c 0xE0 -n 1 --socket-mem=1024,0 --file-prefix testpmd --no-pci \
> --vdev 'net_virtio_user3,mac=00:00:00:00:00:03,path=/var/run/openvswitch/vhost-user3' \
> --vdev 'net_virtio_user4,mac=00:00:00:00:00:04,path=/var/run/openvswitch/vhost-user4' \
> -- -i --burst=64 --disable-hw-vlan --txd=2048 --rxd=2048 --auto-start --coremask=0xc0
> ############################testpmd start script END############################
> 
> The two containers are connected using ovs-docker in the host. I create four vhost-user ports, two of which are for pktgen, the other of which are for testpmd. I connect the vhost-user ports by adding the 
> routes between them. The start script of ovs is:
> 
> ############################ovs-dpdk start script############################
> sudo ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
>                     --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
>                     --private-key=db:Open_vSwitch,SSL,private_key \
>                     --certificate=db:Open_vSwitch,SSL,certificate \
>                     --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
>                     --pidfile --detach
> 
> sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
> 
> sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x02
> 
> sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x04
> 
> sudo ovs-vswitchd --pidfile --detach --log-file=/var/log/openvswitch/vhost-ovs-vswitchd.log
> 
> sudo /usr/local/share/openvswitch/scripts/ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start
> ############################ovs-dpdk start script END############################
> 
> When I start the pktgen-dpdk to send packets, there are something wrong.
> 
> First, I set the packet size to 64Bytes, there are some big packets (more than 64Bytes). I set the rate to 10%, 300 packets, and get:
> 
> ###########################CMD shot#######################
> Ports 0-1 of 2   <Main Page>  Copyright (c) <2010-2017>, Intel Corporation
>  Flags:Port      :   P--------------:0   P--------------:1
> Link State        :       <UP-10000-FD>       <UP-10000-FD>     ----TotalRate----
> Pkts/s Max/Rx     :               256/0               300/0                 556/0
>       Max/Tx     :               300/0               300/0                 600/0
> MBits/s Rx/Tx     :                 0/0                 0/0                   0/0
> Broadcast         :                   0                   0
> Multicast         :                   0                   0
>  64 Bytes        :                1104                1148
>  65-127          :                  64                 152
>  128-255         :                   0                   0
>  256-511         :                   0                   0
>  512-1023        :                   0                   0
>  1024-1518       :                   0                   0
> Runts/Jumbos      :                 0/0                 0/0
> Errors Rx/Tx      :                 0/0                 0/0
> Total Rx Pkts     :                1168                1300
>      Tx Pkts     :                1300                1300
>      Rx MBs      :                   0                   0
>      Tx MBs      :                   1                   1
> ARP/ICMP Pkts     :                 0/0                 0/0
>                  :
> Pattern Type      :             abcd...             abcd...
> Tx Count/% Rate   :            300 /10%            300 /10%
> PktSize/Tx Burst  :           64 /   64           64 /   64
> Src/Dest Port     :         1234 / 5678         1234 / 5678
> Pkt Type:VLAN ID  :     IPv4 / TCP:0001     IPv4 / TCP:0001
> 802.1p CoS        :                   0                   0
> ToS Value:        :                   0                   0
>  - DSCP value    :                   0                   0
>  - IPP  value    :                   0                   0
> Dst  IP Address   :         192.168.1.1         192.168.0.1
> Src  IP Address   :      192.168.0.1/24      192.168.1.1/24
> Dst MAC Address   :   00:00:00:00:00:02   00:00:00:00:00:01
> Src MAC Address   :   00:00:00:00:00:01   00:00:00:00:00:02
> VendID/PCI Addr   :   0000:0000/00:00.0   0000:0000/00:00.0
> 
> -- Pktgen Ver: 3.4.9 (DPDK 17.05.2)  Powered by DPDK --------------------------
> ###########################CMD shot END#######################
> 
> There ought to be no packets greater than 64Bytes, but there exist.
> 
> Second, I reset the configuration ("rst") and try to start send packets continuously. However, the pktgen works few seconds and stop sending packets, with output:
> 
> 
> ###########################CMD shot#######################
> Ports 0-1 of 2   <Main Page>  Copyright (c) <2010-2017>, Intel Corporation
>  Flags:Port      :   P--------------:0   P--------------:1
> Link State        :       <UP-10000-FD>       <UP-10000-FD>     ----TotalRate----
> Pkts/s Max/Rx     :            176288/0            146016/0              308224/0
>       Max/Tx     :           1344832/0            767520/0             1535040/0
> MBits/s Rx/Tx     :                 0/0                 0/0                   0/0
> Broadcast         :                   0                   0
> Multicast         :                   0                   0
>  64 Bytes        :               15872               15104
>  65-127          :               50368               61248
>  128-255         :               44096               62848
>  256-511         :               51840               93216
>  512-1023        :               63264              151456
>  1024-1518       :               51936              126240
> Runts/Jumbos      :                 0/0                 0/0
> Errors Rx/Tx      :                 0/0                 0/0
> Total Rx Pkts     :              277376              510112
>      Tx Pkts     :             4529248             1162368
>      Rx MBs      :                1215                2701
>      Tx MBs      :              665276               57380
> ARP/ICMP Pkts     :                 0/0                 0/0
>                  :
> Pattern Type      :             abcd...             abcd...
> Tx Count/% Rate   :       Forever /100%       Forever /100%
> PktSize/Tx Burst  :           64 /   64           64 /   64
> Src/Dest Port     :         1234 / 5678         1234 / 5678
> Pkt Type:VLAN ID  :     IPv4 / TCP:0001     IPv4 / TCP:0001
> 802.1p CoS        :                   0                   0
> ToS Value:        :                   0                   0
>  - DSCP value    :                   0                   0
>  - IPP  value    :                   0                   0
> Dst  IP Address   :         192.168.1.1         192.168.0.1
> Src  IP Address   :      192.168.0.1/24      192.168.1.1/24
> Dst MAC Address   :   00:00:00:00:00:02   00:00:00:00:00:01
> Src MAC Address   :   00:00:00:00:00:01   00:00:00:00:00:02
> VendID/PCI Addr   :   0000:0000/00:00.0   0000:0000/00:00.0
> 
> -- Pktgen Ver: 3.4.9 (DPDK 17.05.2)  Powered by DPDK --------------------------
> ###########################CMD shot END#######################
> 
> Pktgen is stuck at this setting. There are a lot of large packets!
> 
> I check the log of ovs-dpdk and get:
> 
> #############################ovs-dpdk log#############################
> 2019-06-23T07:51:39.349Z|00022|netdev_dpdk(pmd8)|WARN|Dropped 5803564 log messages in last 102581 seconds (most recently, 102576 seconds ago) due to excessive rate
> 2019-06-23T07:51:39.349Z|00023|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00024|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00025|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00026|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00027|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00028|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00029|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00030|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00031|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00032|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00033|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00034|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00035|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00036|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00037|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00038|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00039|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00040|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00041|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00042|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> #############################ovs-dpdk log END#############################
> Thank you for share your ideas.
> 
> 
> Best wishes,
> Xia Rui

Regards,
Keith



More information about the dev mailing list