[dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes

Take Ceara dumitru.ceara at gmail.com
Tue Jul 26 10:57:29 CEST 2016
Previous message: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes
Next message: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi Beilei,

On Tue, Jul 26, 2016 at 10:47 AM, Zhang, Helin <helin.zhang at intel.com> wrote:
> Hi Ceara
>
> For testpmd command line, txqflags = 0xf01 should be set for receiving packets which needs more than one mbufs.
> I am not sure if it is helpful for you here. Please have a try!
>

Just tried, and it doesn't really help:
testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i
--coremask=0xffff0 --rxq=2 --txq=2 --mbuf-size 1152 --txpkts 1024
--enable-rx-cksum --rss-udp --txqflags 0xf01

  src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
length=1024 - nb_segs=1 - RSS hash=0x0 - RSS queue=0x0 - (outer) L2
type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001
DIP=C0A8000A - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2
type: Unknown - Inner L3 type: Unknown - Inner L4 type: Unknown
 - Receive queue=0x0
  PKT_RX_RSS_HASH

As I was saying in my previous email the problem is that the RSS is
set in the last mbuf instead of the first:

http://dpdk.org/browse/dpdk/tree/drivers/net/i40e/i40e_rxtx.c#n1438

Even worse, the last rxm mbuf was already freed if it only contained
the CRC which had to be stripped:

http://dpdk.org/browse/dpdk/tree/drivers/net/i40e/i40e_rxtx.c#n1419

Regards,
Dumitru


> Regards,
> Helin
>
>> -----Original Message-----
>> From: Take Ceara [mailto:dumitru.ceara at gmail.com]
>> Sent: Tuesday, July 26, 2016 4:38 PM
>> To: Xing, Beilei <beilei.xing at intel.com>
>> Cc: Zhang, Helin <helin.zhang at intel.com>; Wu, Jingjing <jingjing.wu at intel.com>;
>> dev at dpdk.org
>> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs
>> for some RX mbuf sizes
>>
>> Hi Beilei,
>>
>> On Mon, Jul 25, 2016 at 12:04 PM, Take Ceara <dumitru.ceara at gmail.com>
>> wrote:
>> > Hi Beilei,
>> >
>> > On Mon, Jul 25, 2016 at 5:24 AM, Xing, Beilei <beilei.xing at intel.com> wrote:
>> >> Hi,
>> >>
>> >>> -----Original Message-----
>> >>> From: Take Ceara [mailto:dumitru.ceara at gmail.com]
>> >>> Sent: Friday, July 22, 2016 8:32 PM
>> >>> To: Xing, Beilei <beilei.xing at intel.com>
>> >>> Cc: Zhang, Helin <helin.zhang at intel.com>; Wu, Jingjing
>> >>> <jingjing.wu at intel.com>; dev at dpdk.org
>> >>> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for
>> >>> XL710/X710 NICs for some RX mbuf sizes
>> >>>
>> >>> I was using the test-pmd "txonly" implementation which sends fixed
>> >>> UDP packets from 192.168.0.1:1024 -> 192.168.0.2:1024.
>> >>>
>> >>> I changed the test-pmd tx_only code so that it sends traffic with
>> >>> incremental destination IP: 192.168.0.1:1024 -> [192.168.0.2,
>> >>> 192.168.0.12]:1024
>> >>> I also dumped the source and destination IPs in the "rxonly"
>> >>> pkt_burst_receive function.
>> >>> Then I see that packets are indeed sent to different queues but the
>> >>> mbuf->hash.rss value is still 0.
>> >>>
>> >>> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i
>> >>> --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024
>> >>> --enable-rx-cksum --rss-udp
>> >>>
>> >>> [...]
>> >>>
>> >>>  - Receive queue=0xf
>> >>>   PKT_RX_RSS_HASH
>> >>>   src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
>> >>> length=1024 - nb_segs=1 - RSS queue=0xa - (outer) L2 type: ETHER -
>> >>> (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A80006 -
>> >>> (outer)
>> >>> L4 type: UDP - Tunnel type: Unknown - RSS hash=0x0 - Inner L2 type:
>> >>> Unknown - RSS queue=0xf - RSS queue=0x7 - (outer) L2 type: ETHER -
>> >>> (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A80007 -
>> >>> (outer)
>> >>> L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner
>> >>> L3 type: Unknown - Inner L4 type: Unknown
>> >>>  - Receive queue=0x7
>> >>>   PKT_RX_RSS_HASH
>> >>>   src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - (outer) L2 type:
>> >>> ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A80009
>> >>> -
>> >>> type=0x0800 - length=1024 - nb_segs=1 - Inner L3 type: Unknown -
>> >>> Inner
>> >>> L4 type: Unknown - RSS hash=0x0 - (outer) L4 type: UDP - Tunnel type:
>> >>> Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - RSS
>> >>> queue=0x7 - Inner L4 type: Unknown
>> >>>
>> >>> [...]
>> >>>
>> >>> testpmd> stop
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 1/Queue= 0
>> -------
>> >>>   RX-packets: 0              TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 1 -> TX Port= 1/Queue= 1
>> -------
>> >>>   RX-packets: 59             TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 1/Queue= 2
>> -------
>> >>>   RX-packets: 48             TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 1/Queue= 3
>> -------
>> >>>   RX-packets: 0              TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 4 -> TX Port= 1/Queue= 4
>> -------
>> >>>   RX-packets: 59             TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 5 -> TX Port= 1/Queue= 5
>> -------
>> >>>   RX-packets: 0              TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 6 -> TX Port= 1/Queue= 6
>> -------
>> >>>   RX-packets: 0              TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 7 -> TX Port= 1/Queue= 7
>> -------
>> >>>   RX-packets: 48             TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 8 -> TX Port= 1/Queue= 8
>> -------
>> >>>   RX-packets: 0              TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue= 9 -> TX Port= 1/Queue= 9
>> -------
>> >>>   RX-packets: 59             TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue=10 -> TX Port= 1/Queue=10
>> -------
>> >>>   RX-packets: 48             TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue=11 -> TX Port= 1/Queue=11
>> -------
>> >>>   RX-packets: 0              TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue=12 -> TX Port= 1/Queue=12
>> -------
>> >>>   RX-packets: 59             TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue=13 -> TX Port= 1/Queue=13
>> -------
>> >>>   RX-packets: 0              TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue=14 -> TX Port= 1/Queue=14
>> -------
>> >>>   RX-packets: 0              TX-packets: 32             TX-dropped: 0
>> >>>   ------- Forward Stats for RX Port= 0/Queue=15 -> TX Port= 1/Queue=15
>> -------
>> >>>   RX-packets: 48             TX-packets: 32             TX-dropped: 0
>> >>>   ---------------------- Forward statistics for port 0  ----------------------
>> >>>   RX-packets: 428            RX-dropped: 84            RX-total: 512
>> >>>   TX-packets: 0              TX-dropped: 0             TX-total: 0
>> >>>
>> >>> --------------------------------------------------------------------
>> >>> --------
>> >>>
>> >>>   ---------------------- Forward statistics for port 1  ----------------------
>> >>>   RX-packets: 0              RX-dropped: 0             RX-total: 0
>> >>>   TX-packets: 512            TX-dropped: 0             TX-total: 512
>> >>>
>> >>> --------------------------------------------------------------------
>> >>> --------
>> >>>
>> >>>   +++++++++++++++ Accumulated forward statistics for all
>> >>> ports+++++++++++++++
>> >>>   RX-packets: 428            RX-dropped: 84            RX-total: 512
>> >>>   TX-packets: 512            TX-dropped: 0             TX-total: 512
>> >>>
>> >>>
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> +++++++++++++++
>> >>>
>> >>> I checked all the RSS hash values for all the 10 different incoming
>> >>> streams and they're all 0. Also, the fact that the traffic is
>> >>> actually distributed seems to suggest that RSS itself is working but
>> >>> the mbuf hash field is (I guess) either not written or corrupted.
>> >>>
>> >>
>> >> I tried to reproduce the problem with the same steps you used on 16.04 and
>> 16.07, but I really didn't replicate it.
>> >> I think you can try follow ways:
>> >> 1. apply the patch I told you last time and check if the problem still exists.
>> >
>> > I applied the changes in the patch manually to 16.04. The RSS=0
>> > problem still exists while the FDIR issue is fixed.
>> >
>> >> 2. update the codebase and check if the problem still exists.
>> >
>> > I updated the codebase to the latest version on
>> > http://dpdk.org/git/dpdk. I still see the RSS=0 issue.
>> >
>> >> 3. disable vector when you run testpmd, and check if the problem still exists.
>> >
>> > I recompiled the latest dpdk code with
>> > CONFIG_RTE_LIBRTE_I40E_INC_VECTOR=n and the RSS=0 issue is still
>> > there.
>> >
>> > My current command line is:
>> > ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i
>> > --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024
>> > --rss-udp
>> >
>> > Not sure if relevant but I'm running kernel 4.2.0-27:
>> > $ uname -a
>> > Linux jspg2 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22
>> > 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>> >
>> > Is there anything else that might help you identify the cause of the problem?
>> >
>> > Thanks,
>> > Dumitru
>>
>> After some debugging in the i40e DPDK driver I figured out the problem When
>> receiving packets with i40e_recv_scattered_pkts, which gets called in my case
>> because the incoming packet is bigger than 1 full mbuf (the 4 bytes CRC goes in
>> the second mbuf of the chain), the pkt_flags, hash, etc are set only when
>> processing the last mbuf in the packet chain. However, when the hash.rss field is
>> set, instead of setting it in the first mbuf of the packet it gets set in the current
>> mbuf (rxm). This can also cause a lot of unpredictable behavior if the last mbuf
>> only contained the CRC that was stripped as rxm would have already been freed
>> by then. The line I'm referring to is:
>>
>> if (pkt_flags & PKT_RX_RSS_HASH)
>>         rxm->hash.rss =
>>                 rte_le_to_cpu_32(rxd.wb.qword0.hi_dword.rss);
>>
>> I changed it to setting the rss field in first_seg instead of rxm and it works fine
>> now.
>>
>> As far as I see this is the only place where we can receive scattered packets and
>> all the other places where the RSS hash is set seem to be fine.
>> Should I submit a proper patch for this or will you do it as you're more familiar to
>> the code?
>>
>> Thanks,
>> Dumitru
>>
>> >
>> >>
>> >>> >
>> >>> >>
>> >>> >> If I use a different mbuf-size, for example 2048, everything is fine:
>> >>> >>
>> >>> >> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i
>> >>> >> --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 2048 --txpkts
>> >>> >> 1024
>> >>> >> -- enable-rx-cksum --rss-udp [...]
>> >>> >> testpmd> set verbose 1
>> >>> >> Change verbose level from 0 to 1
>> >>> >> testpmd> set fwd rxonly
>> >>> >> Set rxonly packet forwarding mode
>> >>> >> testpmd> start tx_first
>> >>> >>   rxonly packet forwarding - CRC stripping disabled - packets/burst=32
>> >>> >>   nb forwarding cores=16 - nb forwarding ports=2
>> >>> >>   RX queues=16 - RX desc=128 - RX free threshold=32
>> >>> >>   RX threshold registers: pthresh=8 hthresh=8 wthresh=0
>> >>> >>   TX queues=16 - TX desc=512 - TX free threshold=32
>> >>> >>   TX threshold registers: pthresh=32 hthresh=0 wthresh=0
>> >>> >>   TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1:
>> >>> >> received
>> >>> >> 32 packets
>> >>> >>   src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
>> >>> >> length=1024 - nb_segs=1 - RSS hash=0x5263c3f2 - RSS queue=0x1 -
>> >>> >> (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN -
>> >>> >> (outer)
>> >>> >> L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown -
>> >>> >> Inner
>> >>> >> L3 type: Unknown - Inner L4 type: Unknown
>> >>> >>  - Receive queue=0x1
>> >>> >>   PKT_RX_RSS_HASH
>> >>> >>
>> >>> >
>> >>> > Did you send the same packet as before to port 0?
>> >>> >
>> >>> >> Another weird thing I noticed is when I run test-pmd without
>> >>> >> --enable-rx- cksum (which is the default mode) then the RSS flag
>> >>> >> doesn get
>> >>> set at all.
>> >>> >> Instead the PKT_RX_FDIR flag gets set. This happens even though
>> >>> >> fdir_mode is set to RTE_FDIR_MODE_NONE in the device
>> >>> >> configuration:
>> >>> >>
>> >>> >> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i
>> >>> >> --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts
>> >>> >> 1024
>> >>> >> --rss- udp [...]
>> >>> >> testpmd> set verbose 1
>> >>> >> Change verbose level from 0 to 1
>> >>> >> testpmd> set fwd rxonly
>> >>> >> Set rxonly packet forwarding mode
>> >>> >> testpmd> start tx_first
>> >>> >>   rxonly packet forwarding - CRC stripping disabled - packets/burst=32
>> >>> >>   nb forwarding cores=16 - nb forwarding ports=2
>> >>> >>   RX queues=16 - RX desc=128 - RX free threshold=32
>> >>> >>   RX threshold registers: pthresh=8 hthresh=8 wthresh=0
>> >>> >>   TX queues=16 - TX desc=512 - TX free threshold=32
>> >>> >>   TX threshold registers: pthresh=32 hthresh=0 wthresh=0
>> >>> >>   TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1:
>> >>> >> received
>> >>> >> 16 packets
>> >>> >>   src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
>> >>> >> length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263
>> >>> >> Unknown packet type
>> >>> >>  - Receive queue=0x1
>> >>> >>   PKT_RX_FDIR
>> >>> >>   src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 -
>> >>> >> length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263
>> >>> >> Unknown packet type
>> >>> >>  - Receive queue=0x1
>> >>> >>   PKT_RX_FDIR
>> >>> >>
>> >>> >
>> >>> > For this issue, I think following patch can solve your problem,
>> >>> > please apply this
>> >>> patch.
>> >>> > http://dpdk.org/dev/patchwork/patch/13593/
>> >>> >
>> >>>
>> >>> I tried to apply it directly on 16.04 but it can't be applied. I see
>> >>> it's been applied to dpdk-next-net/rel_16_07. Do you happen to have
>> >>> one that would work on the latest stable 16.04 release?
>> >>
>> >> Sorry, I haven't. If there's conflict with R16.04, I think you can resolve it by
>> going through the patch.
>> >>
>> >> Beilei
>> >>
>> >>>
>> >>> Thanks,
>> >>> Dumitru
>> >>>
>> >>> > Beilei
>> >>> >
>> >>> >> Please let me know if there's more debug info that might be of
>> >>> >> interest in troubleshooting the RSS=0 issue.
>> >>> >>
>> >>> >> Thanks,
>> >>> >> Dumitru
>> >>> >>
>> >>> >> > /Beilei
>> >>> >> >
>> >>> >> >> Thanks,
>> >>> >> >> Dumitru
>> >>> >> >>
>> >
>> >
>> >
>> > --
>> > Dumitru Ceara
>>
>>
>>
>> --
>> Dumitru Ceara



-- 
Dumitru Ceara
Previous message: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes
Next message: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the dev mailing list