<div dir="ltr">Hi Lihong,<div> </div><div>I have fixed the NNT server not being called by Jenkins. It should work fine now.</div><div>We did run into an issue where it could not detect huge pages information, but rebooting it again fixed the issue.</div><div>I have also done the same thing to the FVL server. Both of them should be called by Jenkins for testing.</div><div> </div><div>Thanks, Brandon </div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 17, 2020 at 3:00 AM Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div lang="EN-US"> <div> Hi, Brandon I find the NNT server did not be called by the Jenkins, can you help to check? Regards, Ma,lihong From: Brandon Lo [mailto:<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>] Sent: Thursday, April 16, 2020 11:04 PM To: Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>> Cc: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>>; David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>>; Zhang, XuemingX <<a href="mailto:xuemingx.zhang@intel.com" target="_blank">xuemingx.zhang@intel.com</a>>; O'Driscoll, Tim <<a href="mailto:tim.odriscoll@intel.com" target="_blank">tim.odriscoll@intel.com</a>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing <div> Hi Lihong, <div> </div> <div> I saw your changes and applied them manually. </div> <div> This involves me taking the machine down for maintenance on our backend, and then doing a normal `reboot`. </div> <div> </div> <div> Now when I do `cat /proc/cmdline` it shows: </div> <div> `BOOT_IMAGE=/boot/vmlinuz-4.15.0-55-generic root=UUID=3801030b-237d-428e-9e67-d81e12f16308 ro quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable vt.handoff=1` </div> <div> <div> </div> <div> Which is what I think we wanted to achieve. </div> <div> I have now done the `reboot_to_ro` command and it should be working fine for testing. </div> <div> </div> <div> Thanks, </div> <div> Brandon </div> <div> </div> </div> </div> <div> <div> On Wed, Apr 15, 2020 at 11:40 PM Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"> <div> <div> Hi, Brandon <a name="m_1874102666203270993_m_253694866417265958_m_4827830322542245087__MailEndCompose">I use the cmd ‘reboot_to_rw’ to restart server, and edit the file ‘/etc/default/grub’ to reset the </a>isolation cpus, Changes the field ‘isolcpus=1-48 nohz_full=1-48 rcu_nocbs=1-48’ to ‘isolcpus=1-21,49-69,24-37,72-85 nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85’ Detail as below: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable" Then use cmd: update-grub to make the configuration effective. Then reboot server use ‘reboot_to_ro’, but I can’t find the change in cmdline ( cat /proc/cmdline). Can you help me to check it? Thanks Lihong From: Brandon Lo [mailto:<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>] Sent: Thursday, April 16, 2020 3:31 AM To: Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>> Cc: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>>; David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>>; Zhang, XuemingX <<a href="mailto:xuemingx.zhang@intel.com" target="_blank">xuemingx.zhang@intel.com</a>>; O'Driscoll, Tim <<a href="mailto:tim.odriscoll@intel.com" target="_blank">tim.odriscoll@intel.com</a>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing <div> Hi Lihong, <div> </div> <div> I think any time is fine to make these changes. </div> <div> </div> <div> Please use the 'reboot_to_rw' command once you have logged on, this will make sure your changes are saved. </div> <div> It will also disable testing momentarily while you make your changes. </div> <div> </div> <div> Once you're finished, you can reboot it with 'reboot_to_ro'. </div> <div> Please let me know when you're done so I can make sure everything is working as intended. </div> <div> We could also do this change for you if needed. </div> <div> </div> <div> Thanks, </div> <div> Brandon </div> </div> <div> <div> On Wed, Apr 15, 2020 at 5:39 AM Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt"> <div> <div> Hi, Brandon I find the grub configuration of isolation cpu miss the logic cores which at some thread. For example: If the server cpus layout as below, and want to config the isolation cpu from 1-20 The config of isolation should is ‘isolcpus=1-20,49-68’ <img border="0" width="396" height="509" id="gmail-m_1874102666203270993gmail-m_253694866417265958gmail-m_4827830322542245087gmail-m_767458348881989906Picture_x0020_1" src="cid:17188b069144ce8e91" alt="cid:1717f4fa8006917eb1"> So, I want to change the grub configuration of the isolation cpus. From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ Can you help me check which time is suite to do this change ? Regards, Ma,lihong From: Brandon Lo [mailto:<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>> Cc: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>>; David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>>; Zhang, XuemingX <<a href="mailto:xuemingx.zhang@intel.com" target="_blank">xuemingx.zhang@intel.com</a>>; O'Driscoll, Tim <<a href="mailto:tim.odriscoll@intel.com" target="_blank">tim.odriscoll@intel.com</a>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing <div> Hi Lihong, <div> </div> <div> I have changed the baselines to reflect the new expected values. </div> <div> The performance tests should work as expected and pass. </div> <div> </div> <div> We will email again in the future if we come across any problems. </div> <div> Feel free to email us as well if you would like to make any other changes. </div> <div> </div> <div> Thank you for all your help </div> </div> <div> <div> On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt"> <div> <div> Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: <table border="0" cellspacing="0" cellpadding="0" width="0" style="width:191pt;border-collapse:collapse"> <tbody> <tr style="height:17.4pt"> <td width="75" nowrap style="width:48pt;border:1pt solid windowtext;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> pkt_size </td> <td width="69" nowrap style="width:48pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> trd/rxd </td> <td width="130" nowrap style="width:95pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> expected_value </td> </tr> <tr style="height:0.2in"> <td width="75" nowrap valign="bottom" style="width:48pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt;height:0.2in"> 64 </td> <td width="69" nowrap valign="bottom" style="width:48pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 512 </td> <td width="130" nowrap valign="bottom" style="width:95pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 52.562 </td> </tr> <tr style="height:0.2in"> <td width="75" nowrap valign="bottom" style="width:48pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt;height:0.2in"> 64 </td> <td width="69" nowrap valign="bottom" style="width:48pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 2048 </td> <td width="130" nowrap valign="bottom" style="width:95pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 41.439 </td> </tr> </tbody> </table> FVL: <table border="0" cellspacing="0" cellpadding="0" width="0" style="width:191pt;border-collapse:collapse"> <tbody> <tr style="height:17.4pt"> <td width="75" nowrap style="width:48pt;border:1pt solid windowtext;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> pkt_size </td> <td width="69" nowrap style="width:48pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> trd/rxd </td> <td width="130" nowrap style="width:95pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> expected_value </td> </tr> <tr style="height:0.2in"> <td width="75" nowrap valign="bottom" style="width:48pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt;height:0.2in"> 64 </td> <td width="69" nowrap valign="bottom" style="width:48pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 512 </td> <td width="130" nowrap valign="bottom" style="width:95pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 59.608 </td> </tr> <tr style="height:0.2in"> <td width="75" nowrap valign="bottom" style="width:48pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt;height:0.2in"> 64 </td> <td width="69" nowrap valign="bottom" style="width:48pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 2048 </td> <td width="130" nowrap valign="bottom" style="width:95pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 47.73 </td> </tr> </tbody> </table> For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: <table border="0" cellspacing="0" cellpadding="0" width="0" style="width:238.25pt;border-collapse:collapse"> <tbody> <tr style="height:17.4pt"> <td width="75" nowrap style="width:56.4pt;border:1pt solid windowtext;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> pkt_size </td> <td width="69" nowrap style="width:51.6pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> trd/rxd </td> <td width="174" nowrap style="width:130.25pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> expected_value </td> </tr> <tr style="height:0.2in"> <td width="75" nowrap valign="bottom" style="width:56.4pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt;height:0.2in"> 64 </td> <td width="69" nowrap valign="bottom" style="width:51.6pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 512 </td> <td width="174" nowrap valign="bottom" style="width:130.25pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 52.562 / 2.5 * 2.1=44.152 </td> </tr> <tr style="height:0.2in"> <td width="75" nowrap valign="bottom" style="width:56.4pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt;height:0.2in"> 64 </td> <td width="69" nowrap valign="bottom" style="width:51.6pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 2048 </td> <td width="174" nowrap valign="bottom" style="width:130.25pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 41.439 / 2.5 * 2.1=34.809 </td> </tr> </tbody> </table> FVL: <table border="0" cellspacing="0" cellpadding="0" width="0" style="width:238.25pt;border-collapse:collapse"> <tbody> <tr style="height:17.4pt"> <td width="75" nowrap style="width:56.4pt;border:1pt solid windowtext;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> pkt_size </td> <td width="69" nowrap style="width:51.6pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> trd/rxd </td> <td width="174" nowrap style="width:130.25pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;background:rgb(0,102,204);padding:0in 5.4pt;height:17.4pt"> expected_value </td> </tr> <tr style="height:0.2in"> <td width="75" nowrap valign="bottom" style="width:56.4pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt;height:0.2in"> 64 </td> <td width="69" nowrap valign="bottom" style="width:51.6pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 512 </td> <td width="174" nowrap valign="bottom" style="width:130.25pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 59.608 / 2.5 * 2.1=50.071 </td> </tr> <tr style="height:0.2in"> <td width="75" nowrap valign="bottom" style="width:56.4pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt;height:0.2in"> 64 </td> <td width="69" nowrap valign="bottom" style="width:51.6pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 2048 </td> <td width="174" nowrap valign="bottom" style="width:130.25pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in"> 47.73 / 2.5 * 2.1=40.093 </td> </tr> </tbody> </table> Regards, Ma,lihong From: Brandon Lo [mailto:<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> Cc: David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>>; Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>>; Zhang, XuemingX <<a href="mailto:xuemingx.zhang@intel.com" target="_blank">xuemingx.zhang@intel.com</a>>; O'Driscoll, Tim <<a href="mailto:tim.odriscoll@intel.com" target="_blank">tim.odriscoll@intel.com</a>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing <div> Hi Zhaoyan, <div> </div> <div> To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. </div> <div> This command will also disable any testing on the machine. </div> <div> </div> <div> To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. </div> <div> I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. </div> <div> </div> <div> After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. </div> <div> </div> <div> Thank you </div> </div> <div> <div> On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt"> <div> <div> Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen <div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt"> <div> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> From: Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> Cc: David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>>; Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>>; Zhang, XuemingX <<a href="mailto:xuemingx.zhang@intel.com" target="_blank">xuemingx.zhang@intel.com</a>>; O'Driscoll, Tim <<a href="mailto:tim.odriscoll@intel.com" target="_blank">tim.odriscoll@intel.com</a>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing </div> </div> <div> Hi Zhaoyan, <div> </div> <div> Currently, we have a system in place that resets any changes made while testing is enabled for a machine. </div> <div> If you would like, I can disable testing and allow you to make permanent changes. </div> <div> </div> <div> I can also reset the baseline of Intel 10G test performance once you make these changes. </div> <div> Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. </div> <div> </div> <div> Thanks </div> </div> <div> <div> On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt"> <div> <div> Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen <div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt"> <div> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> From: Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> Cc: David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>>; Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>>; Zhang, XuemingX <<a href="mailto:xuemingx.zhang@intel.com" target="_blank">xuemingx.zhang@intel.com</a>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing </div> </div> <div> Hi Zhaoyan, <div> </div> <div> I have enabled the 10G Intel machine for testing. </div> <div> If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. </div> <div> Please feel free to let me know if you need anything. </div> <div> </div> <div> Thank you </div> </div> <div> <div> On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt"> <div> <div> Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen <div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt"> <div> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> From: Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> Cc: David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>>; Ma, LihongX <<a href="mailto:lihongx.ma@intel.com" target="_blank">lihongx.ma@intel.com</a>>; Zhang, XuemingX <<a href="mailto:xuemingx.zhang@intel.com" target="_blank">xuemingx.zhang@intel.com</a>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing </div> </div> <div> Hi Zhaoyan, <div> </div> <div> Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. </div> <div> Should I disable the 40G Intel machine for you to make changes? </div> <div> </div> <div> Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? </div> <div> </div> <div> If everything is ok, I will enable the 10G machine for production testing. </div> <div> </div> <div> Thank you very much </div> </div> <div> <div> On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt"> <div> <div> Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it <ul type="disc"> <li class="MsoNormal" style="color:rgb(31,73,125)"> Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system</li><li class="MsoNormal" style="color:rgb(31,73,125)"> When we debugging on the testbed, found that Trex was killed by some one(app)..</li></ul> Please help to check if any other program running on the testbed. Thanks a lot. <div> Regards, Zhaoyan Chen </div> <div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt"> <div> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> From: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> Cc: David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>>; Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing </div> </div> Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen <div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt"> <div> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> From: Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> Cc: David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing </div> </div> <div> Hi Zhaoyan, <div> </div> <div> Have you finished making changes on the Intel machine? </div> <div> I will turn on the machine on March 3rd for testing if you do not have any issues with it. </div> <div> Please let me know if you need anything else. </div> <div> </div> <div> Thanks </div> </div> <div> <div> On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt"> <div> <div> Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen <div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt"> <div> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> From: Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>> Cc: Chen, Zhaoyan <<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>>; <a href="mailto:dpdklab@iol.unh.edu" target="_blank">dpdklab@iol.unh.edu</a>; Lincoln Lavoie <<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>>; Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan <<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>>; Xu, Qian Q <<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing </div> </div> <div> Hi Zhaoyan, <div> </div> <div> How is the current status of the Intel 82599ES? </div> <div> Were there any configuration changes made to fix performance issues? </div> <div> </div> <div> Thanks </div> </div> <div> <div> On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt"> <div> Hi David, <div> </div> <div> This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. </div> <div> I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. </div> <div> </div> <div> Thanks </div> </div> <div> <div> On Tue, Mar 10, 2020 at 5:33 AM David Marchand <<a href="mailto:david.marchand@redhat.com" target="_blank">david.marchand@redhat.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt"> On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). <a href="https://lab.dpdk.org/results/dashboard/patchsets/9900/" target="_blank">https://lab.dpdk.org/results/dashboard/patchsets/9900/</a> But this time, it is with the 40G Intel nic test. -- David Marchand </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </div> </div> </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </div> </div> </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </div> </div> </blockquote> </div> <div> </div> -- <div> <div> <div> <div> Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 <a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a> <a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a> </div> </div> </div> </div> </div> </div> </blockquote></div> <div> </div>-- <div dir="ltr"><div dir="ltr"><div><div dir="ltr">Brandon LoUNH InterOperability Laboratory21 Madbury Rd, Suite 100, Durham, NH 03824<a href="mailto:blo@iol.unh.edu" target="_blank">blo@iol.unh.edu</a><a href="http://www.iol.unh.edu/" target="_blank">www.iol.unh.edu</a></div></div></div></div>