[dpdk-ci] New 100G Broadcom NIC Ubuntu

Brandon Lo blo at iol.unh.edu
Thu Oct 22 21:03:59 CEST 2020


Hi Ajit,

Just checking in to make sure this issue is not dropped.
I have tested the 100G NIC running TREX/DTS again, it still seems that
the 100G NIC is not able to use TREX as the console is outputting
"Connection lost - Subscriber timeout: no data from TRex server."
However, I know that some developers are busy for DPDK so I understand
if you are a bit pressed on time currently.

Thanks,
Brandon

On Tue, Sep 1, 2020 at 2:12 PM Ajit Khaparde <ajit.khaparde at broadcom.com> wrote:
>
> Hi Brandon,
> Thanks for this.
> I have been stretched thin lately, and this will surely help.
>
> Thanks
> Ajit
>
> On Tue, Sep 1, 2020 at 11:09 AM Brandon Lo <blo at iol.unh.edu> wrote:
>>
>> Hi Ajit,
>>
>> I'm going to re-enable the Broadcom 25G NIC in production performance
>> and functional testing.
>>
>> If you would like to do testing/debugging on the machine, such as on
>> the 100G NIC, then please disable these two pipelines on our Jenkins:
>>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/
>>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/
>>
>> You will need to sign in using the same credentials as your VPN.
>>
>> This will prevent any overlapping of DTS/TREX runs which is known to
>> cause errors.
>> Please let me know if you run into any issues.
>>
>> Thanks,
>> Brandon
>>
>> On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde
>> <ajit.khaparde at broadcom.com> wrote:
>> >
>> > Not yet Brandon.
>> > But I believe I might have to look at what trex is doing to make progress.
>> > I will let you know if I need any help.
>> >
>> > Thanks
>> > Ajit
>> >
>> > On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo <blo at iol.unh.edu> wrote:
>> >>
>> >> Hi Ajit,
>> >>
>> >> Did you find anything of interest on the machine?
>> >> I could help out if needed.
>> >>
>> >> Thanks,
>> >> Brandon
>> >>
>> >> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde
>> >> <ajit.khaparde at broadcom.com> wrote:
>> >> >
>> >> > Hi Brandon,
>> >> > I do see some issues while running trex on this setup.
>> >> > I will have to dig further. I will let you know once I find something or may be a fix.
>> >> >
>> >> > Thanks
>> >> > Ajit
>> >> >
>> >> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo at iol.unh.edu> wrote:
>> >> >>
>> >> >> Hi Ajit,
>> >> >>
>> >> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
>> >> >> testpmd (through an ssh session from io --> rhea) to catch information
>> >> >> as a DUT.
>> >> >>
>> >> >> The only commands that I run (on io) are:
>> >> >> cd /opt/dts
>> >> >> export DTS_CFG_FOLDER='conf_100g'
>> >> >> ./dts -s
>> >> >>
>> >> >> The rest is managed by DTS itself.
>> >> >> The issue occurs when DTS is trying to send packets to rhea from io on
>> >> >> the new 100G NIC.
>> >> >> It seems to happen after it tries to measure throughput using TREX's API.
>> >> >>
>> >> >> Thanks,
>> >> >> Brandon
>> >> >>
>> >> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
>> >> >> <ajit.khaparde at broadcom.com> wrote:
>> >> >> >
>> >> >> > Hi Brandon,
>> >> >> > I was trying to see what exactly is happening on the setup and what is causing the problem.
>> >> >> > But I will need your help to use the proper code and commands.
>> >> >> >
>> >> >> > I believe you run the testpmd command on io. And the trex is executed on rhea?
>> >> >> > Can you point me to the location of the code and the steps you are following while running the test?
>> >> >> >
>> >> >> > Thanks
>> >> >> > Ajit
>> >> >> >
>> >> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <ajit.khaparde at broadcom.com> wrote:
>> >> >> >>
>> >> >> >> Hi Brandon,
>> >> >> >> I haven't. I tried to login as well. But I had some issues doing it from the office.
>> >> >> >> I just have to remind myself to try it again once I get home before I connect to the company VPN.
>> >> >> >> Thanks for checking in. I will try to update you as soon as I can.
>> >> >> >>
>> >> >> >> Thanks
>> >> >> >> Ajit
>> >> >> >>
>> >> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo at iol.unh.edu> wrote:
>> >> >> >>>
>> >> >> >>> Hi Ajit,
>> >> >> >>>
>> >> >> >>> I'm just checking in; have you heard of any updates on this issue?
>> >> >> >>>
>> >> >> >>> Thanks,
>> >> >> >>> Brandon
>> >> >> >>>
>> >> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo at iol.unh.edu> wrote:
>> >> >> >>>>
>> >> >> >>>> Hi Ajit,
>> >> >> >>>>
>> >> >> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC.
>> >> >> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing.
>> >> >> >>>>
>> >> >> >>>> If you want to replicate it, here are the steps that I ran:
>> >> >> >>>>
>> >> >> >>>> (on io) cd /opt/dts
>> >> >> >>>> export DTS_CFG_FOLDER='conf_100g'
>> >> >> >>>>
>> >> >> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file
>> >> >> >>>>
>> >> >> >>>> ./dts -s
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> Thanks for your help,
>> >> >> >>>> Brandon
>> >> >> >>>>
>> >> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde at broadcom.com> wrote:
>> >> >> >>>>>
>> >> >> >>>>> Hi Brandon,
>> >> >> >>>>> No, I haven't seen or heard this before.
>> >> >> >>>>> But I will try to have someone run it again.
>> >> >> >>>>>
>> >> >> >>>>> Just to make sure -
>> >> >> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup?
>> >> >> >>>>> Or is it the system that is running testpmd where you are running into the issue?
>> >> >> >>>>>
>> >> >> >>>>> Thanks
>> >> >> >>>>> Ajit
>> >> >> >>>>>
>> >> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo at iol.unh.edu> wrote:
>> >> >> >>>>>>
>> >> >> >>>>>> Hi Ajit,
>> >> >> >>>>>>
>> >> >> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC.
>> >> >> >>>>>> The current configuration uses trex version v2.82.
>> >> >> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf.
>> >> >> >>>>>>
>> >> >> >>>>>> The output looks like this when it reaches a test case:
>> >> >> >>>>>>
>> >> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>> >> >> >>>>>>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512 --rxd=512
>> >> >> >>>>>>>                       dut.rhea: start
>> >> >> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >> >> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
>> >> >> >>>>>>>     case_obj()
>> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core
>> >> >> >>>>>>>     self.perf_test(self.nb_ports)
>> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
>> >> >> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> >> >> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission
>> >> >> >>>>>>>     self._conn.reset(ports=self._ports)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >> >> >>>>>>>     ret = f(*args, **kwargs)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>> >> >> >>>>>>>     self.clear_stats(ports)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >> >> >>>>>>>     ret = f(*args, **kwargs)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats
>> >> >> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common
>> >> >> >>>>>>>     raise TRexError(rc)
>> >> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
>> >> >> >>>>>> Have you ran into this issue before?
>> >> >> >>>>>>
>> >> >> >>>>>> Thanks,
>> >> >> >>>>>> Brandon
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> --
>> >> >> >>>>
>> >> >> >>>> Brandon Lo
>> >> >> >>>>
>> >> >> >>>> UNH InterOperability Laboratory
>> >> >> >>>>
>> >> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >> >> >>>>
>> >> >> >>>> blo at iol.unh.edu
>> >> >> >>>>
>> >> >> >>>> www.iol.unh.edu
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>>
>> >> >> >>> Brandon Lo
>> >> >> >>>
>> >> >> >>> UNH InterOperability Laboratory
>> >> >> >>>
>> >> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >> >> >>>
>> >> >> >>> blo at iol.unh.edu
>> >> >> >>>
>> >> >> >>> www.iol.unh.edu
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >>
>> >> >> Brandon Lo
>> >> >>
>> >> >> UNH InterOperability Laboratory
>> >> >>
>> >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >> >>
>> >> >> blo at iol.unh.edu
>> >> >>
>> >> >> www.iol.unh.edu
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Brandon Lo
>> >>
>> >> UNH InterOperability Laboratory
>> >>
>> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >>
>> >> blo at iol.unh.edu
>> >>
>> >> www.iol.unh.edu
>>
>>
>>
>> --
>>
>> Brandon Lo
>>
>> UNH InterOperability Laboratory
>>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>
>> blo at iol.unh.edu
>>
>> www.iol.unh.edu



-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo at iol.unh.edu

www.iol.unh.edu


More information about the ci mailing list