<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Hi Brandon,</div><div class="gmail_default" style="font-family:tahoma,sans-serif">Thanks for this.</div><div class="gmail_default" style="font-family:tahoma,sans-serif">I have been stretched thin lately, and this will surely help.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Thanks</div><div class="gmail_default" style="font-family:tahoma,sans-serif">Ajit</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 1, 2020 at 11:09 AM Brandon Lo <<a href="mailto:blo@iol.unh.edu">blo@iol.unh.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Ajit,<br>
<br>
I'm going to re-enable the Broadcom 25G NIC in production performance<br>
and functional testing.<br>
<br>
If you would like to do testing/debugging on the machine, such as on<br>
the 100G NIC, then please disable these two pipelines on our Jenkins:<br>
<a href="https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/" rel="noreferrer" target="_blank">https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/</a><br>
<a href="https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/" rel="noreferrer" target="_blank">https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/</a><br>
<br>
You will need to sign in using the same credentials as your VPN.<br>
<br>
This will prevent any overlapping of DTS/TREX runs which is known to<br>
cause errors.<br>
Please let me know if you run into any issues.<br>
<br>
Thanks,<br>
Brandon<br>
<br>
On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde<br>
<<a href="mailto:ajit.khaparde@broadcom.com" target="_blank">ajit.khaparde@broadcom.com</a>> wrote:<br>
><br>
> Not yet Brandon.<br>
> But I believe I might have to look at what trex is doing to make progress.<br>
> I will let you know if I need any help.<br>
><br>
> Thanks<br>
> Ajit<br>
><br>
> On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> wrote:<br>
>><br>
>> Hi Ajit,<br>
>><br>
>> Did you find anything of interest on the machine?<br>
>> I could help out if needed.<br>
>><br>
>> Thanks,<br>
>> Brandon<br>
>><br>
>> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde<br>
>> <<a href="mailto:ajit.khaparde@broadcom.com" target="_blank">ajit.khaparde@broadcom.com</a>> wrote:<br>
>> ><br>
>> > Hi Brandon,<br>
>> > I do see some issues while running trex on this setup.<br>
>> > I will have to dig further. I will let you know once I find something or may be a fix.<br>
>> ><br>
>> > Thanks<br>
>> > Ajit<br>
>> ><br>
>> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> wrote:<br>
>> >><br>
>> >> Hi Ajit,<br>
>> >><br>
>> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs<br>
>> >> testpmd (through an ssh session from io --> rhea) to catch information<br>
>> >> as a DUT.<br>
>> >><br>
>> >> The only commands that I run (on io) are:<br>
>> >> cd /opt/dts<br>
>> >> export DTS_CFG_FOLDER='conf_100g'<br>
>> >> ./dts -s<br>
>> >><br>
>> >> The rest is managed by DTS itself.<br>
>> >> The issue occurs when DTS is trying to send packets to rhea from io on<br>
>> >> the new 100G NIC.<br>
>> >> It seems to happen after it tries to measure throughput using TREX's API.<br>
>> >><br>
>> >> Thanks,<br>
>> >> Brandon<br>
>> >><br>
>> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde<br>
>> >> <<a href="mailto:ajit.khaparde@broadcom.com" target="_blank">ajit.khaparde@broadcom.com</a>> wrote:<br>
>> >> ><br>
>> >> > Hi Brandon,<br>
>> >> > I was trying to see what exactly is happening on the setup and what is causing the problem.<br>
>> >> > But I will need your help to use the proper code and commands.<br>
>> >> ><br>
>> >> > I believe you run the testpmd command on io. And the trex is executed on rhea?<br>
>> >> > Can you point me to the location of the code and the steps you are following while running the test?<br>
>> >> ><br>
>> >> > Thanks<br>
>> >> > Ajit<br>
>> >> ><br>
>> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <<a href="mailto:ajit.khaparde@broadcom.com" target="_blank">ajit.khaparde@broadcom.com</a>> wrote:<br>
>> >> >><br>
>> >> >> Hi Brandon,<br>
>> >> >> I haven't. I tried to login as well. But I had some issues doing it from the office.<br>
>> >> >> I just have to remind myself to try it again once I get home before I connect to the company VPN.<br>
>> >> >> Thanks for checking in. I will try to update you as soon as I can.<br>
>> >> >><br>
>> >> >> Thanks<br>
>> >> >> Ajit<br>
>> >> >><br>
>> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> wrote:<br>
>> >> >>><br>
>> >> >>> Hi Ajit,<br>
>> >> >>><br>
>> >> >>> I'm just checking in; have you heard of any updates on this issue?<br>
>> >> >>><br>
>> >> >>> Thanks,<br>
>> >> >>> Brandon<br>
>> >> >>><br>
>> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> wrote:<br>
>> >> >>>><br>
>> >> >>>> Hi Ajit,<br>
>> >> >>>><br>
>> >> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC.<br>
>> >> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing.<br>
>> >> >>>><br>
>> >> >>>> If you want to replicate it, here are the steps that I ran:<br>
>> >> >>>><br>
>> >> >>>> (on io) cd /opt/dts<br>
>> >> >>>> export DTS_CFG_FOLDER='conf_100g'<br>
>> >> >>>><br>
>> >> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file<br>
>> >> >>>><br>
>> >> >>>> ./dts -s<br>
>> >> >>>><br>
>> >> >>>><br>
>> >> >>>> Thanks for your help,<br>
>> >> >>>> Brandon<br>
>> >> >>>><br>
>> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <<a href="mailto:ajit.khaparde@broadcom.com" target="_blank">ajit.khaparde@broadcom.com</a>> wrote:<br>
>> >> >>>>><br>
>> >> >>>>> Hi Brandon,<br>
>> >> >>>>> No, I haven't seen or heard this before.<br>
>> >> >>>>> But I will try to have someone run it again.<br>
>> >> >>>>><br>
>> >> >>>>> Just to make sure -<br>
>> >> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup?<br>
>> >> >>>>> Or is it the system that is running testpmd where you are running into the issue?<br>
>> >> >>>>><br>
>> >> >>>>> Thanks<br>
>> >> >>>>> Ajit<br>
>> >> >>>>><br>
>> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> wrote:<br>
>> >> >>>>>><br>
>> >> >>>>>> Hi Ajit,<br>
>> >> >>>>>><br>
>> >> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC.<br>
>> >> >>>>>> The current configuration uses trex version v2.82.<br>
>> >> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf.<br>
>> >> >>>>>><br>
>> >> >>>>>> The output looks like this when it reaches a test case:<br>
>> >> >>>>>><br>
>> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512<br>
>> >> >>>>>>> dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1 --file-prefix=dpdk_11307_20200804160513 -- -i --portmask=0x3 --txd=512 --rxd=512<br>
>> >> >>>>>>> dut.rhea: start<br>
>> >> >>>>>>> TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):<br>
>> >> >>>>>>> File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case<br>
>> >> >>>>>>> case_obj()<br>
>> >> >>>>>>> File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core<br>
>> >> >>>>>>> self.perf_test(self.nb_ports)<br>
>> >> >>>>>>> File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test<br>
>> >> >>>>>>> _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)<br>
>> >> >>>>>>> File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput<br>
>> >> >>>>>>> self._prepare_transmission(stream_ids=stream_ids)<br>
>> >> >>>>>>> File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission<br>
>> >> >>>>>>> self._conn.reset(ports=self._ports)<br>
>> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2<br>
>> >> >>>>>>> ret = f(*args, **kwargs)<br>
>> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset<br>
>> >> >>>>>>> self.clear_stats(ports)<br>
>> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2<br>
>> >> >>>>>>> ret = f(*args, **kwargs)<br>
>> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats<br>
>> >> >>>>>>> self._clear_stats_common(ports, clear_global, clear_xstats)<br>
>> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common<br>
>> >> >>>>>>> raise TRexError(rc)<br>
>> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://<a href="http://127.0.0.1:4501" rel="noreferrer" target="_blank">127.0.0.1:4501</a><br>
>> >> >>>>>><br>
>> >> >>>>>><br>
>> >> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: <a href="https://github.com/cisco-system-traffic-generator/trex-core/issues/147" rel="noreferrer" target="_blank">https://github.com/cisco-system-traffic-generator/trex-core/issues/147</a>.<br>
>> >> >>>>>> Have you ran into this issue before?<br>
>> >> >>>>>><br>
>> >> >>>>>> Thanks,<br>
>> >> >>>>>> Brandon<br>
>> >> >>>><br>
>> >> >>>><br>
>> >> >>>><br>
>> >> >>>> --<br>
>> >> >>>><br>
>> >> >>>> Brandon Lo<br>
>> >> >>>><br>
>> >> >>>> UNH InterOperability Laboratory<br>
>> >> >>>><br>
>> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824<br>
>> >> >>>><br>
>> >> >>>> <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a><br>
>> >> >>>><br>
>> >> >>>> <a href="http://www.iol.unh.edu" rel="noreferrer" target="_blank">www.iol.unh.edu</a><br>
>> >> >>><br>
>> >> >>><br>
>> >> >>><br>
>> >> >>> --<br>
>> >> >>><br>
>> >> >>> Brandon Lo<br>
>> >> >>><br>
>> >> >>> UNH InterOperability Laboratory<br>
>> >> >>><br>
>> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824<br>
>> >> >>><br>
>> >> >>> <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a><br>
>> >> >>><br>
>> >> >>> <a href="http://www.iol.unh.edu" rel="noreferrer" target="_blank">www.iol.unh.edu</a><br>
>> >><br>
>> >><br>
>> >><br>
>> >> --<br>
>> >><br>
>> >> Brandon Lo<br>
>> >><br>
>> >> UNH InterOperability Laboratory<br>
>> >><br>
>> >> 21 Madbury Rd, Suite 100, Durham, NH 03824<br>
>> >><br>
>> >> <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a><br>
>> >><br>
>> >> <a href="http://www.iol.unh.edu" rel="noreferrer" target="_blank">www.iol.unh.edu</a><br>
>><br>
>><br>
>><br>
>> --<br>
>><br>
>> Brandon Lo<br>
>><br>
>> UNH InterOperability Laboratory<br>
>><br>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824<br>
>><br>
>> <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a><br>
>><br>
>> <a href="http://www.iol.unh.edu" rel="noreferrer" target="_blank">www.iol.unh.edu</a><br>
<br>
<br>
<br>
-- <br>
<br>
Brandon Lo<br>
<br>
UNH InterOperability Laboratory<br>
<br>
21 Madbury Rd, Suite 100, Durham, NH 03824<br>
<br>
<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a><br>
<br>
<a href="http://www.iol.unh.edu" rel="noreferrer" target="_blank">www.iol.unh.edu</a><br>
</blockquote></div>