[dpdk-ci] Intel performance test is failing

[Jeremy Plsek]

It's possible this is happening more frequently because of enabling
overlayfs.
We also changed how the baseline is retrieved because of the overlayfs
system, but this issue was happening before this change.
I think this started happening after a few months after upgrading to Ubuntu
18.04.

Here are the recent results (login to view and download the artifacts):
https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19

Here are some results that were "off" (see the "Run 1" tab):
https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19

And here are some older tests where this occurred:
https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19

I also found some tests where it happened on the 40g platform:
https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18
https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18
(older, before enabling overlayfs):
https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18
https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18


On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan <zhaoyan.chen at intel.com>
wrote:

> Hi, Jeremy and Lincoln,
>
>
>
> Did you change anything on the Intel testbed? DTS code or testbed
> environment?
>
> And could you send me the detailed test logs? We will do a quick check.
>
>
>
> Thanks
>
>
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Lincoln Lavoie <lylavoie at iol.unh.edu>
> *Sent:* Friday, January 24, 2020 4:29 AM
> *To:* Jeremy Plsek <jplsek at iol.unh.edu>
> *Cc:* Thomas Monjalon <thomas at monjalon.net>; ci at dpdk.org; Tu, Lijuan <
> lijuan.tu at intel.com>; Chen, Zhaoyan <zhaoyan.chen at intel.com>; Xu, Qian Q <
> qian.q.xu at intel.com>
> *Subject:* Re: [dpdk-ci] Intel performance test is failing
>
>
>
> In the "off cases," is that specific to that hardware, or does it happen
> on all hardware variants from time-to-time?
>
>
>
> On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek at iol.unh.edu> wrote:
>
> On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas at monjalon.net>
> wrote:
> >
> > Hi,
> >
> > The test iol-intel-Performance is failing on all patches
> > for a couple of days.
> > It seems to be due to a perf drop with a 10G device
> >
> > Can we disable this test until we understand what happens?
> >
> > We need also to understand how to avoid such failure,
> > and how to monitor and disable it faster.
> >
> > Thanks
> >
> >
> Hi Thomas,
>
> I've disabled it for now until someone from the Intel team can look
> into it (cc'd).
>
> I've also seem problems where the results are completely off
> (-15mpps), and I've been rerunning those when they happen. I'm hoping
> those can be resolved too.
>
> --
> Jeremy Plsek
> UNH InterOperability Laboratory
>
>
>
>
> --
>
> *Lincoln Lavoie*
>
> Senior Engineer, Broadband Technologies
>
> 21 Madbury Rd., Ste. 100, Durham, NH 03824
>
> lylavoie at iol.unh.edu
>
> https://www.iol.unh.edu
>
> +1-603-674-2755 (m)
>
> <https://www.iol.unh.edu/>
>


-- 
Jeremy Plsek
UNH InterOperability Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mails.dpdk.org/archives/ci/attachments/20200131/7d72d5c8/attachment.html>