<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hi Adam,<br>
<br>
> Now that our test results are in line with yours, we can
begin looking into setting up the production environment.<br>
<br>
Please, let me know if you need any help with it or waiting for an
input from me.<br>
<br>
Regards,<br>
Andrew.<br>
<br>
On 10/10/23 17:09, Adam Hassick wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAC-YWqgpc_ByrpUJxLa1nGtj7C+Kd34NR2aRYqZ_s0jAMQTMnw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
Thank you for taking a look at our log. Netplan was
attempting to run DHCP on our test links, and
additionally I discovered that the NIC firmware was
transmitting LLDP packets, causing tests to fail in
the same way. Now that these problems have been
fixed, our pass rate on the XL710 is approximately
91%. Now that our test results are in line with
yours, we can begin looking into setting up the
production environment.</div>
</div>
<br>
</div>
First, is it possible to run the test agent on ARM hosts?
Our ARM testbeds have the best topology for running this
test suite, with separate tester and DUT servers.<br>
<br>
</div>
We are testing this test suite on two x86 development
servers using the test suite's recommended server topology.
In contrast, our existing x86 production testbeds which run
DTS have a single server topology. This single server has
both the tester NIC and the device under test NIC installed,
with NUMA node separation between TRex and DPDK. We're going
to test running the two test agent processes on the
single-server testbeds if we cannot run this on ARM. Is
there any reason you can think of that would prevent this
setup from working?<br>
<br>
</div>
<div>Once we figure out where this can live in production,
then we will begin setting up log storage, Jenkins
integration, and Bublik.<br>
</div>
<div><br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Oct 5, 2023 at 6:25 AM
Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
> Do these default to vfio-pci?<br>
<br>
Yes, vfio-pci is the default.<br>
However, it does not work in the case of Mellanox which
uses bifurcated driver. It should mlx5_core for Mellanox
NICs.<br>
<br>
> Here is the text log from a run on our Intel XL710
NICs, with the expected result profile set to the X710.<br>
<br>
It is hard to analyze all tests using text logs, but I
definitely see one common problem. Tests receive
unexpected packets and fail because of it.<br>
Tests are written very strict from this point of view and
it brought fruits in the past when HW had bugs.<br>
Are DUT and tester connected back-to-back on tested
interfaces or via switch?<br>
If via switch, is it possible to isolate it from
everything else?<br>
If back-to-back, it could be some embedded SW/FW which
regenerates these packet.<br>
I definitely see unexpected DHCP packets.<br>
<br>
> We haven't set up the Jenkins integration yet,
however if this is required to import the logs then we
will prioritize that.<br>
<br>
Unfortunately manual runs do not generate all artifacts
required to import logs. However, we have almost solved it
right now. Hopefully we'll finalize it in a day or two.
I'll let you know when these changes are available.<br>
<br>
Regards,<br>
Andrew.<br>
<br>
On 10/4/23 16:48, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
Ok, that makes sense. I don't see
TE_ENV_H1/H2_DPDK_DRIVER set anywhere in the
default configurations for the Intel X710. Do
these default to vfio-pci?<br>
</div>
<div>We have IOMMU enabled on our development
testbed, and should be able to bind vfio-pci.<br>
</div>
<div>Here is the text log from a run on our Intel
XL710 NICs, with the expected result profile set
to the X710. We haven't set up the Jenkins
integration yet, however if this is required to
import the logs then we will prioritize that.<br>
</div>
<div class="gmail_chip gmail_drive_chip"
style="width:396px;height:18px;max-height:18px;background-color:rgb(245,245,245);padding:5px;color:rgb(34,34,34);font-family:arial;font-style:normal;font-weight:bold;font-size:13px;border:1px solid rgb(221,221,221);line-height:1"><a
href="https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web"
style="display:inline-block;max-width:366px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;text-decoration:none;padding:1px 0px;border:medium"
aria-label="log.txt.tar.gz" target="_blank"
moz-do-not-send="true"><img
style="vertical-align: bottom; border: medium;"
src="https://ssl.gstatic.com/docs/doclist/images/icon_10_generic_list.png"
moz-do-not-send="true"> <span dir="ltr"
style="color:rgb(17,85,204);text-decoration:none;vertical-align:bottom">log.txt.tar.gz</span></a><img
style="opacity: 0.55; float: right; display: none;"
moz-do-not-send="true"></div>
<br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Sep 18, 2023
at 11:04 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>On 9/18/23 17:44, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>Hi Andrew and Konstantin,<br>
<br>
</div>
Thank you for adding the tester-dial
feature, this opens up the
possibility for us to do CI
integrated testing in the future.<br>
<br>
</div>
Our Mellanox pass rate is similar to
yours (about ~2400 passing, ~4400
failing), however our Intel pass rates
are far worse.<br>
</div>
<div>I will try running tests on the
XL710 with the trc-tags argument set
and see if it improves the pass rate.<br>
</div>
Another thing I noticed in the results
you uploaded is that the results are
tagged with vfio-pci and not i40e.</div>
<div>Though in the environment dump, the
driver on the test machine and the DUT
are set to use the i40e driver. Is this
important at all?<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
I think it is a misunderstanding here. There are two
kinds of driver in configuration: net driver and
so-called DPDK driver.<br>
Net driver is a Linux kernel network device driver
used on Tester side.<br>
DPDK driver is a Linux kernel driver to bind device
to to use it with DPDK. So, it is NOT a driver
inside DPDK (drivers/net/*).<br>
In the case of bifurcated driver (like mlx5_core) it
is the same in both cases.<br>
In non-bifurcated case DPDK driver is some UIO
driver(vfio-pci, uio-pci-generic or igb_uio).<br>
Some expectations depend on used UIO. For example,
uio-pci-generic do not support many interrupts (used
by usecases/rx_intr test cases).<br>
That's why we care corresponding TRC tag.<br>
<br>
TE_ENV_*_DPDK_DRIVER variables should be vfio-pc in
710's Intel case. Or uio-pci-generic if IOMMU is
turned off on corresponding machines and Linux
distro does not support VFIO no IOMMU mode.<br>
<br>
Andrew.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>There isn't anything preventing us from
pushing our results up to the existing
Bublik instance running at <a
href="http://ts-factory.io"
target="_blank" moz-do-not-send="true">ts-factory.io</a>
that I can think of at the moment.<br>
</div>
<div>We will have to work out how to submit
our results to your Bublik instance in a
controlled and secure manner in that case.<br>
</div>
<div>As far as I know we won't need access
controls for the results themselves. I'll
discuss this with Patrick and will let you
know once we confirm that it's fine.</div>
</div>
<div><br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Sep
18, 2023 at 2:26 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>On 9/18/23 09:23, Konstantin Ushakov
wrote:<br>
</div>
<blockquote type="cite">
<div style="font-family:sans-serif">
<div style="white-space:normal">
<p dir="auto">Hi Andrew,</p>
<p dir="auto">should we always
auto-assign the tags or you don’t do
it since it slows down (by some
seconds) the TE startup?</p>
</div>
</div>
</blockquote>
<br>
Tags are auto-assigned, but I guess it
differs in Adam's case since NIC is a bit
different. Below test will help to
understand if it is the root cause of very
different expectations. If pass rate will be
close to mine, I'll simply update TRC
database to share expectations for mine NIC
and NIC used by Adam.<br>
<br>
<blockquote type="cite">
<div style="font-family:sans-serif">
<div style="white-space:normal">
<p dir="auto">Hi Adam,</p>
<p dir="auto">I think I second the
question from Andrew - happy to help
you with the triage so that we get
to the same baseline. Do you have a
good way for us to share the logs?
I.e. say upload to ts-factory if we
add strict permissions system so
it’s not publishing or any other
way.</p>
<p dir="auto">Thanks, <br>
Konstantin</p>
<br>
<p dir="auto">On 18 Sep 2023, at 9:15,
Andrew Rybchenko wrote:</p>
</div>
<blockquote
style="margin:0px 0px 5px;padding-left:5px;border-left:2px solid rgb(119,119,119);color:rgb(119,119,119)">
<div
id="m_3634070300823225067m_-4480488732519633353m_-792332968217640304733A56D0A-0ED3-47F6-99B4-35C92E41C2DA">
<div>Hi Adam,<br>
<br>
I've uploaded fresh testing
results to <a
href="http://ts-factory.io"
target="_blank"
moz-do-not-send="true">ts-factory.io</a>
[1] to be on the same page.<br>
<br>
I think I know why your and mine
results on Intel 710 series NICs
differ so much. Testing results
expectations database
(dpdk-ethdev-ts/trc/*) is filled
in in terms of TRC tags. I.e.
expectations depends on TRC tags
discovered by helper scripts when
testing is started. These tags
identify various aspects of what
is tested. Ideally expectations
should be written in terms of root
cause of the expected behaviour.
If it is a driver expectations,
driver tag should be used. If it
is HW limitation, tags with PCI
IDs should be used. However, it is
not always easy to classify it
correctly if you're not involved
in driver development. So, in
order case expectations for 710's
Intel are filled in in terms of
PCI IDs. I guess PCI ID differ in
your case and that's why
expectations filled in for my NIC
do not apply to your runs.<br>
<br>
Just try to add the following
option when you run on your 710's
Intel in order to mimic mine and
see if it helps to achieve better
pass rate.<br>
--trc-tag=pci-8086-1572<br>
<br>
BTW, fresh TE tag <span>v1.21.0
has improved algorithm to choose
tests for --tester-dial option.
It should have better coverage
now.</span><br>
<br>
Andrew.<br>
<br>
[1] <a
href="https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1"
target="_blank"
moz-do-not-send="true">https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1</a><br>
<br>
On 9/13/23 18:45, Andrew Rybchenko
wrote:<br>
</div>
<blockquote type="cite">
<div>Hi Adam,<br>
<br>
I've pushed new TE tag v1.20.0
which supported a new
command-line option
--tester-dial=NUM where NUM is
from 0 to 100. it allows to
choose percentage of tests to
run. If you want stable set, you
should pass
--tester-random-seed=0 (or other
integer). It is the first sketch
and we have plans to improve it,
but feedback would be welcome.<br>
<br>
> Is it needed on the tester?<br>
<br>
It is hard to say if it is
strictly required for simple
tests. However, it is better to
update Tester as well, since
performance tests run DPDK on
Tester as well.<br>
<br>
> Are there any other manual
setup steps for these devices
that I might be missing?<br>
<br>
I don't remember anything else.<br>
<br>
I think it is better to get down
to details and take a look at
logs. I'm ready to help with it
and explain what's happening
there. May be it will help to
understand if it is a problem
with setup/configuration.<br>
<br>
Text logs are not very
convenient. Ideally logs should
be imported to bublik, however,
manual runs do not provide all
required artifacts right now
(Jenkins jobs generate all
required artifacts).<br>
Other option is 'tmp_raw_log'
file (should be packed to make
it smaller) which could be
converted to various log
formats.<br>
Would it be OK for you if I
import your logs to bublik at <a
href="http://ts-factory.io"
target="_blank"
moz-do-not-send="true">ts-factory.io</a>?
Or is it a problem that it is
publicly available?<br>
Would it help if we add
authentication and access
control there?<br>
<br>
Andrew.<br>
<br>
On 9/8/23 17:57, Adam Hassick
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
I have a couple
questions about needed
setup of the NICs for
the ethdev test suite.<br>
<br>
</div>
Our MCX5s and XL710s are
failing the checkup
tests. The pass rate
appears to be much worse
on the XL710s (40 of 73
tests failed, 3 passed
unexpectedly).<br>
<br>
</div>
For the XL710s, I've
updated the driver and NVM
versions to match the
minimum supported versions
in the compatibility
matrix found on the DPDK
documentation. This did
not change the failure
rate much.<br>
</div>
For the MCX5s, I've
installed the latest LTS
version of the OFED
bifurcated driver on the
DUT. Is it needed on the
tester?<br>
<br>
</div>
Are there any other manual
setup steps for these devices
that I might be missing?<br>
<div>
<div>
<div>
<div>
<div>
<div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed,
Sep 6, 2023 at 11:00 AM Adam
Hassick <<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>> wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
<div>Yes, I copied
the X710 configs
to set up XL710
configs. I
changed the
environment
variable names
from the X710
suffix to XL710
suffix in the
script, and
forgot to change
them in the
corresponding
environment
file.<br>
</div>
</div>
That fixed the
issue.<br>
<br>
</div>
I got the checkup
tests working on the
XL710 now. Most of
them are failing,
which leads me to
believe this is an
issue with our
testbed. Based on the
DPDK documentation for
i40e, the firmware and
driver versions are
much older than what
DPDK 22.11 LTS and
main prefer, so I'll
try updating those.<br>
<br>
</div>
For now I'm working on
getting the XL710
checkup tests passing,
and will pick up getting
the E810 configured
properly next. I'll let
you know if I run into
any more issues in
relation to the test
engine.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On
Wed, Sep 6, 2023 at
7:36 AM Andrew Rybchenko
<<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
On 9/5/23 18:01,
Adam Hassick wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi
Andrew,<br>
<br>
</div>
The
compilation
warning issue
is now
resolved.
Again, thank
you guys for
fixing this
for us. I can
run the tests
on the
Mellanox CX5s
again, however
I'm running
into a couple
new issues
with running
the prologues
on the Intel
cards.<br>
<br>
</div>
When running
testing on the
Intel XL710s,
I see this
error appear
in the log:<br>
<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">ERROR
prologue
Environment
LIB
14:16:13.650<br>
Too few
networks in
available
configuration
(0) in
comparison
with required
(1)<br>
</blockquote>
<br>
</div>
This seems
like a trivial
configuration
error, perhaps
this is
something I
need to set up
in ts-rigs. I
briefly
searched
through the
examples there
and didn't see
any mention of
how to set up
a network.<br>
</div>
<div>I will
attach this
log just in
case you need
more
information.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Unfortunately logs are
insufficient to
understand it. I've
pushed new tag to TE
v1.19.0 which add log
message with TE_*
environment variables.<br>
Most likely something
is wrong with
variables which are
used as conditions
when available
networks are defined
in
ts-conf/cs/inc.net_cfg_pci_fns.yml:<br>
TE_PCI_INSTANCE_IUT_TST1<br>
TE_PCI_INSTANCE_IUT_TST1a<br>
TE_PCI_INSTANCE_TST1a_IUT<br>
TE_PCI_INSTANCE_TST1_IUT<br>
My guess it that you
change naming a bit,
but script like
ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>There is a
different
error when
running on the
Intel E810s.
It appears to
me like it
starts DPDK,
does some
configuration
inside DPDK
and on the
device, and
then fails to
bring the
device back
up. Since this
error seems
very
non-trivial, I
will also
attach this
log.<br>
</div>
</div>
</div>
</blockquote>
<br>
This one is a bit
simpler. Few lines
after the first ERROR
in log I see the
following:<br>
WARN RCF DPDK
13:06:00.144<br>
ice_program_hw_rx_queue(): currently package doesn't support RXDID (22)<br>
ice_rx_queue_start():
fail to program RX
queue 0<br>
ice_dev_start(): fail
to start Rx queue 0<br>
Device with port_id=0
already stopped<br>
<br>
It is stdout/stderr
from test agent which
runs DPDK. Same logs
in plain format are
available in ta.DPDK
file.<br>
I'm not an expert
here, but I vaguely
remember that E810
requires correct
firmware and DDP to be
loaded.<br>
There is some
information in
dpdk/doc/guides/nics/ice.rst.<br>
<br>
You can try to add
--dev-args=safe-mode-support=1
command-line option
described there.<br>
<br>
Hope it helps,<br>
Andrew.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div><br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
On 8/31/23
22:38, Adam
Hassick wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>Hi
Andrew,<br>
</div>
<div><br>
I have one
additional
question as
well: Does the
test engine
support
running tests
on two ARMv8
test agents?</div>
<div><br>
</div>
<div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">1.
We'll sort out
warnings this
week. Thanks
for heads up.<br>
</blockquote>
<div><br>
</div>
<div>Great.
Let me know
when that's
fixed.</div>
</div>
</div>
</blockquote>
<br>
Done. We also
fixed a number
of warnings in
TE.<br>
Also we fixed
root test
package name
to be
consistent
with the
repository
name.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>Support
for old LTS
branches was
dropped some
time ago, but
in the future
it is
definitely
possible to
keep it for
new LTS
branches. I
think 22.11 is
supported, but
I'm not sure
about older
LTS releases.</div>
</blockquote>
<div><br>
</div>
<div>Good to
know.<br>
<div> <br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2.
You can add
command-line
option
--sanity to
run tests
marked with
TEST_HARNESS_SANITY
requirement
(see
dpdk-ethdev-ts/scripts/run.sh
and grep
TEST_HARNESS_SANITY
dpdk-ethdev-ts
to see which
tests are
marked). Yes,
there is a
space for
terminology
improvement
here. We'll do
it.<br>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
Done. Now it
is called
--checkup.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Also it takes
a lot of time
because of
failures and
tests which
wait for some
timeout.<br>
</blockquote>
</div>
<div><br>
</div>
<div>That
makes sense to
me. We'll use
the time to
complete tests
on virtio or
the Intel
devices as a
reference for
how long the
tests really
take to
complete.<br>
</div>
<div>We will
explore the
possibility of
periodically
running the
sanity tests
for patches.<br>
</div>
</div>
</div>
</blockquote>
<br>
I'll
double-check
and let you
know how long
entire TS runs
on Intel X710,
E810, Mellanox
CX5 and virtio
net. Just to
ensure that
time observed
in your case
looks the
same.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div> <br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The
test harness
can provide
coverage
reports based
on gcov, but
I'm not sure
what you mean
by a "dial" to
control test
coverage.
Provided
reports are
rather for
human to
analyze.<br>
</blockquote>
</div>
<div><br>
</div>
<div>The
general idea
is to have
some kind of
parameter on
the test
suite, which
could be an
integer
ranging from
zero to ten,
that controls
how many tests
are run based
on how
important the
test is.<br>
<br>
</div>
<div>Similar
to how some
command line
interfaces
provide a
verbosity
level
parameter
(some number
of "-v"
arguments) to
control the
importance of
the
information in
the log.<br>
</div>
The verbosity
level zero
only prints
very important
log messages,
while ten
prints
everything.<br>
</div>
<div><br>
In much the
same manner as
above, this
"dial"
parameter
controls what
tests are run
and with what
parameters
based on how
important
those tests
and test
parameter
combinations
are.<br>
Coverage Level
zero tells the
suite to run a
very basic set
of important
tests, with
minimal
parameterization.
This mode
would take
only ~5-10
minutes to
run.<br>
In contrast,
Coverage Level
ten includes
all the edge
cases, every
combination of
test
parameters,
everything the
test suite can
do, which
takes the
normal several
hours to run.<br>
The values 1 -
9 are between
those two
extremes,
allowing the
user to get a
gradient of
test coverage
in the results
and to limit
the running
time.<br>
<br>
</div>
Then we could,
for example,
run the
"run.sh" with
a level of 2
or 3 for
incoming
patches that
need quick
results, and
with a level
of 10 for the
less often run
periodic tests
performed on
main or LTS
branches.<br>
</div>
</blockquote>
<br>
Understood
now. Thanks a
lot for the
idea. We'll
discuss it and
come back.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>3. Yes,
really many
tests on
Mellanox CX5
NICs report
unexpected
testing
results.
Unfortunately
it is time
consuming to
fill in
expectations
database since
it is
necessary to
analyze
testing
results and
classify if it
is a bug or
just
acceptable
behaviour
aspect.<br>
<br>
Bublik allows
to compare
results of two
runs. It is
useful for
human, but
still not good
for
automation.<br>
<br>
I have local
patch for mlx5
driver which
reports Tx
ring size
maximum. It
makes pass
rate higher.
It is a
problem for
test harness
that mlx5 does
not report
limits right
now.<br>
<br>
Pass rate on
Intel X710 is
about 92% on
my test rig.
Pass rate on
virtio net is
99% right now
and could be
done 100%
easily (just
one thing to
fix in
expectations).<br>
<br>
I think logs
storage setup
is essential
for logs
analysis. Of
course, you
can request
HTML logs when
you run tests
(--log-html=html) or generate after run using
dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a browser, but
logs storage
makes it more
convenient.<br>
</div>
</div>
</blockquote>
<div><br>
We are
interested in
setting up
Bublik,
potentially as
an
externally-facing
component,
once we have
our process of
running the
test suite
stabilized.</div>
<div>Once we
are able to
run the test
suite again,
I'll see what
the pass rate
is on our
other
hardware.<br>
Good to know
that it isn't
an issue with
our dev
testbed
causing the
high fail
rate.</div>
</div>
<div>
<div><br>
</div>
<div>For Intel
hardware, we
have an XL710
and an Intel
E810-C in our
development
testbed.
Although they
are slightly
different
devices,
ideally the
pass rate will
be identical
or similar. I
have yet to
set up a VM
pair for
virtio, but we
will soon.<br>
</div>
<div><br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Latest
version of
test-environment
has examples
of our CGI
scripts which
we use for log
storage (see
tools/log_server/README.md).<br>
<br>
Also all bits
for Jenkins
setup are
available. See
dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
ts-rigs-sample.<br>
</blockquote>
</div>
<div><br>
</div>
<div>Jenkins
integration,
setting up
production rig
configurations, and permanent log storage will be our next steps once I
am able to run
the tests
again.<br>
</div>
<div>Unless
there is an
easy way to
have meson not
pass "-Werror"
into GCC. Then
I would be
able to run
the test
suite.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Hopefully it
is resolved
now.<br>
<br>
I thought a
bit more about
your usecase
for Jenkins.
I'm not 100%
sure that
existing
pipelines are
convenient for
your usecase.<br>
Fill free to
ask questions
when you are
on it.<br>
<br>
Thanks,<br>
Andrew.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
<div><br>
</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div><br>
On 8/29/23
17:02, Adam
Hassick wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi
Andrew,<br>
<br>
</div>
That fix seems
to have
resolved the
issue, thanks
for the quick
turnaround
time on that
patch.<br>
</div>
<div>Now that
we have the
RCF timeout
issue
resolved,
there are a
few other
questions and
issues that we
have about the
tests
themselves.</div>
<br>
</div>
<div>1. The
test suite
fails to build
with a couple
warnings.<br>
</div>
<div><br>
</div>
<div>Below is
the stderr log
from
compilation:<br>
</div>
<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">FAILED:
<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
cc
-Ilib/76b5a35@@ts_dpdk_pmd@sta
-Ilib
-I../../lib
-I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
-fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
-Winvalid-pch
-Werror -g
-D_GNU_SOURCE
-O0 -ggdb
-Wall -W -fPIC
-MD -MQ '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-MF '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
-o '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-c
../../lib/dpdk_pmd_ts.c<br>
../../lib/dpdk_pmd_ts.c: In function
‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
no format
arguments
[-Werror=format-security]<br>
5577 | rc
=
te_kvpair_add(result,
buf, mode);<br>
| ^~<br>
cc1: all
warnings being
treated as
errors<br>
ninja: build
stopped:
subcommand
failed.<br>
ninja:
Entering
directory `.'<br>
FAILED: <a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
cc
-Ilib/76b5a35@@ts_dpdk_pmd@sta
-Ilib
-I../../lib
-I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
-fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
-Winvalid-pch
-Werror -g
-D_GNU_SOURCE
-O0 -ggdb
-Wall -W -fPIC
-MD -MQ '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-MF '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
-o '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-c
../../lib/dpdk_pmd_ts.c<br>
../../lib/dpdk_pmd_ts.c: In function
‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
no format
arguments
[-Werror=format-security]<br>
5577 | rc
=
te_kvpair_add(result,
buf, mode);<br>
| ^~<br>
cc1: all
warnings being
treated as
errors<br>
</blockquote>
<div>
<div>
<div><br>
</div>
<div>This
error wasn't
occurring last
week, which
was the last
time I ran the
tests.<br>
</div>
<div>The TE
host and the
DUT have GCC
v9.4.0
installed, and
the tester has
GCC v11.4.0
installed, if
this
information is
helpful.<br>
</div>
<div><br>
</div>
<div>2. On the
Mellanox CX5s,
there are over
6,000 tests
run, which
collectively
take around 9
hours. Is it
possible, and
would it make
sense, to
lower the test
coverage and
have the test
suite run
faster?<br>
<br>
</div>
<div>For some
context, we
run immediate
testing on
incoming
patches for
DPDK main and
development
branches, as
well as
periodic test
runs on the
main, stable,
and LTS
branches.<br>
</div>
<div>For us to
consider
including this
test suite as
part of our
immediate
testing on
patches, we
would have to
reduce the
test coverage
to the most
important
tests.<br>
This is
primarily to
reduce the
testing time
to, for
example, less
than 30
minutes.
Testing on
patches can't
take too long
because the
lab can
receive
numerous
patches each
day, which
each require
individual
testing runs.<br>
<br>
</div>
<div>At what
frequency we
run these
tests, and on
what, still
needs to be
discussed with
the DPDK
community, but
it would be
nice to know
if the test
suite had a
"dial" to
control the
testing
coverage.<br>
</div>
<div><br>
</div>
<div>3. We see
a lot of test
failures on
our Mellanox
CX5 NICs.
Around 2,300
of ~6,600
tests passed.
Is there
anything we
can do to
diagnose these
test failures?<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
I've pushed
the fix in
main branch
and a new tag
v1.18.1. It
should solve
the problem
with IPv6
address from
DNS.<br>
<br>
Andrew.<br>
<br>
On 8/29/23
00:05, Andrew
Rybchenko
wrote:<br>
</div>
<blockquote
type="cite">
<div>Hi Adam,<br>
<br>
> Does the
test engine
prefer to use
IPv6 over IPv4
for initiating
the RCF
connection to
the test bed
hosts? And if
so, is there a
way to force
it to use
IPv4?<br>
<br>
Brilliant
idea. If DNS
returns both
IPv4 and IPv6
addresses in
your case, I
guess it is
the root cause
of the
problem.<br>
Of course, it
is TE problem
since I see
really weird
code in
lib/comm_net_engine/comm_net_engine.c
line 135.<br>
<br>
I've pushed
fix to the
branch
user/arybchik/fix_ipv4_only
in
ts-factory/test-environment
repository.
Please, try.<br>
<br>
It is late
night fix with
minimal
testing and no
review. I'll
pass it
through review
process
tomorrow and<br>
hopefully it
will be
released in
one-two days.<br>
<br>
Andrew.<br>
<br>
On 8/28/23
18:02, Adam
Hassick wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi
Andrew,<br>
<br>
</div>
We have yet to
notice a
distinct
pattern with
the failures.
Sometimes, the
RCF will start
and connect
without issue
a few times in
a row before
failing to
connect again.
Once the issue
begins to
occur, neither
rebooting all
of the hosts
(test engine
VM, tester,
IUT) or
deleting all
of the build
directories
(suites,
agents, inst)
and rebooting
the hosts
afterward
resolves the
issue. When it
begins working
again seems
very arbitrary
to us.<br>
<br>
</div>
<div>I do
usually try to
terminate the
test engine
with Ctrl+C,
but when it
hangs while
trying to
start RCF,
that does not
work.<br>
</div>
<div><br>
</div>
<div>Does the
test engine
prefer to use
IPv6 over IPv4
for initiating
the RCF
connection to
the test bed
hosts? And if
so, is there a
way to force
it to use
IPv4?<br>
<br>
</div>
<div> - Adam<br>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>> I'll
double-check
test engine on
Ubuntu 20.04
and Ubuntu
22.04.<br>
<br>
Done. It works
fine for me
without any
issues.<br>
<br>
Have you
noticed any
pattern when
it works or
does not work?<br>
May be it is a
problem of not
clean state
after
termination?<br>
Does it work
fine the first
time after
DUTs reboot?<br>
How do you
terminate
testing? It
should be done
using Ctrl+C
in terminal
where you
execute run.sh
command.<br>
In this case
it should
shutdown
gracefully and
close all test
agents and
engine
applications.<br>
<br>
(I'm trying to
understand why
you've seen
many test
agent
processes. It
should not
happen.)<br>
<br>
Andrew.<br>
<br>
On 8/25/23
17:41, Andrew
Rybchenko
wrote:<br>
</div>
<blockquote
type="cite">
<div>On
8/25/23 17:06,
Adam Hassick
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>Hi
Andrew,<br>
<br>
</div>
Two of our
systems (the
Test Engine
runner and the
DUT host) are
running Ubuntu
20.04 LTS,
however this
morning I
noticed that
the tester
system (the
one having
issues) is
running Ubuntu
22.04 LTS.<br>
</div>
<div>This
could be the
source of the
problem. I
encountered a
dependency
issue trying
to run the
Test Engine on
22.04 LTS, so
I downgraded
the system.
Since the
tester is also
the host
having
connection
issues, I will
try
downgrading
that system to
20.04, and see
if that
changes
anything.<br>
</div>
</div>
</blockquote>
<br>
Unlikely, but
who knows. We
run tests
(DUTs) on
Ubuntu 20.04,
Ubuntu 22.04,
Ubuntu 22.10,
Ubuntu 23.04,
Debian 11 and
Fedora 38
every night.<br>
Right now
Debian 11 is
used for test
engine in
nightly
regressions.<br>
<br>
I'll
double-check
test engine on
Ubuntu 20.04
and Ubuntu
22.04.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>I did try
passing in the
"--vg-rcf"
argument to
the run.sh
script of the
test suite
after
installing
valgrind, but
there was no
additional
output that I
saw.<br>
</div>
</div>
</blockquote>
<br>
Sorry, I
should
valgrind
output should
be in
valgrind.te_rcf
(direction
where you run
test engine).<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I will
try pulling in
the changes
you've pushed
up, and will
see if that
fixes
anything.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hello
Adam,<br>
<br>
On 8/24/23
23:54, Andrew
Rybchenko
wrote:<br>
</div>
<blockquote
type="cite">I'd
like to try to
repeat the
problem
locally. Which
Linux distro
is running on
test engine
and agents?<br>
<br>
In fact I know
one problem
with Debian 12
and Fedora 38
and we have<br>
patch in
review to fix
it, however,
the behaviour
is different
in<br>
this case, so
it is unlike
the same
problem.<br>
</blockquote>
<br>
I've just
published a
new tag which
fixes known
test engine
side problems
on Debian 12
and Fedora 38.<br>
<br>
<blockquote
type="cite"><br>
One more idea
is to install
valgrind on
the test
engine host
and<br>
run with
option
--vg-rcf to
check if
something
weird is
happening.<br>
<br>
What I don't
understand
right now is
why I see just
one failed
attempt<br>
to connect in
your log.txt
and then
Logger
shutdown after
9 minutes.<br>
<br>
Andrew.<br>
<br>
On 8/24/23
23:29, Adam
Hassick wrote:<br>
<blockquote
type="cite"> >
Is there any
firewall in
the network or
on test hosts
which could
block incoming
TCP connection
to the port
23571 <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host
where you run
test engine?<br>
<br>
Our test
engine host
and the
testbed are on
the same
subnet. The
connection
does work
sometimes.<br>
<br>
> If
behaviour the
same on the
next try and
you see that
test agent is
kept running,
could you
check using<br>
><br>
> #
netstat -tnlp<br>
><br>
> that
Test Agent is
listening on
the port and
try to
establish TCP
connection
from test
agent using<br>
><br>
> $ telnet
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank"
moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a> <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571 <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a><br>
><br>
> and
check if TCP
connection
could be
established.<br>
<br>
I was able to
replicate the
same behavior
again, where
it hangs while
RCF is trying
to start.<br>
Running this
command, I see
this in the
output:<br>
<br>
tcp 0
0 <a
href="http://0.0.0.0:23571" target="_blank" moz-do-not-send="true">0.0.0.0:23571</a>
<a
href="http://0.0.0.0:23571" target="_blank" moz-do-not-send="true"><http://0.0.0.0:23571></a>
0.0.0.0:*
LISTEN
18599/ta<br>
<br>
So it seems
like it is
listening on
the correct
port.<br>
Additionally,
I was able to
connect to the
Tester machine
from our Test
Engine host
using telnet.
It printed the
PID of the
process once
the connection
was opened.<br>
<br>
I tried
running the
"ta"
application
manually on
the command
line, and it
didn't print
anything at
all.<br>
Maybe the
issue is
something on
the Test
Engine side.<br>
<br>
On Thu, Aug
24, 2023 at
2:35 PM Andrew
Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote:<br>
<br>
Hi Adam,<br>
<br>
> On
the tester
host (which
appears to be
the Peer
agent), there<br>
are four
processes that
I see running,
which look
like the test<br>
agent
processes.<br>
<br>
Before the
next try I'd
recommend to
kill these
processes.<br>
<br>
Is there
any firewall
in the network
or on test
hosts which
could<br>
block
incoming TCP
connection to
the port 23571<br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host<br>
where you
run test
engine?<br>
<br>
If
behaviour the
same on the
next try and
you see that
test agent is<br>
kept
running, could
you check
using<br>
<br>
# netstat
-tnlp<br>
<br>
that Test
Agent is
listening on
the port and
try to
establish TCP<br>
connection
from test
agent using<br>
<br>
$ telnet <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank"
moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a><br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571<br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a><br>
<br>
and check
if TCP
connection
could be
established.<br>
<br>
Another
idea is to
login Tester
under root as
testing does,
get<br>
start TA
command from
the log and
try it by
hands without
-n and<br>
remove
extra
escaping.<br>
<br>
# sudo
PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1<br>
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=<br>
<br>
Hopefully
in this case
test agent
directory
remains in the
/tmp and<br>
you don't
need to copy
it as testing
does.<br>
May be
output could
shed some
light on
what's going
on.<br>
<br>
Andrew.<br>
<br>
On 8/24/23
17:30, Adam
Hassick wrote:<br>
<blockquote
type="cite">
Hi Andrew,<br>
<br>
This is
the output
that I see in
the terminal
when this
failure<br>
occurs,
after the test
agent binaries
build and the
test engine<br>
starts:<br>
<br>
Platform
default build
- pass<br>
Simple RCF
consistency
check
succeeded<br>
--->>>
Starting
Logger...done<br>
--->>>
Starting
RCF...rcf_net_engine_connect():
Connection
timed<br>
out <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu:23571</a><br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a><br>
<br>
Then, it
hangs here
until I kill
the "te_rcf"
and "te_tee"<br>
processes.
I let it hang
for around 9
minutes.<br>
<br>
On the
tester host
(which appears
to be the Peer
agent), there
are<br>
four
processes that
I see running,
which look
like the test
agent<br>
processes.<br>
<br>
ta.Peer is
an empty file.
I've attached
the log.txt
from this run.<br>
<br>
- Adam<br>
<br>
On Thu,
Aug 24, 2023
at 4:22 AM
Andrew
Rybchenko<br>
<<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a><br>
<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote:<br>
<br>
Hi
Adam,<br>
<br>
Yes,
TE_RCFUNIX_TIMEOUT
is in seconds.
I've
double-checked<br>
that
it goes to
'copy_timeout'
in
ts-conf/rcf.conf.<br>
Description in
in
doc/sphinx/pages/group_te_engine_rcf.rst<br>
says
that
copy_timeout
is in seconds
and
implementation
in<br>
lib/rcfunix/rcfunix.c
passes the
value to
select()
tv_sec.<br>
Theoretically
select() could
be interrupted
by signal, but
I<br>
think
it is unlikely
here.<br>
<br>
I'm
not sure that
I understand
what do you
mean by RCF<br>
connection
timeout. Does
it happen on
TE startup
when RCF<br>
starts
test agents.
If so,
TE_RCFUNIX_TIMEOUT
could help. Or<br>
does
it happen when
tests are in
progress, e.g.
in the middle<br>
of a
test. If so,
TE_RCFUNIX_TIMEOUT
is unrelated
and most<br>
likely
either host
with test
agent dies or
test agent
itself<br>
crashes. It
would be
easier for me
if classify it
if you share<br>
text
log (log.txt,
full or just
corresponding
fragment with<br>
some
context). Also
content of
ta.DPDK or
ta.Peer file<br>
depending on
which agent
has problems
could shed
some light.<br>
Corresponding
files contain
stdout/stderr
of test
agents.<br>
<br>
Andrew.<br>
<br>
On
8/23/23 17:45,
Adam Hassick
wrote:<br>
<blockquote
type="cite">
Hi Andrew,<br>
<br>
I've
set up a test
rig repository
here, and have
created<br>
configurations
for our
development
testbed based
off of the<br>
examples.<br>
We've
been able to
get the test
suite to run
manually on<br>
Mellanox CX5
devices once.<br>
However, we
are running
into an issue
where, when
RCF starts,<br>
the
RCF connection
times out very
frequently. We
aren't sure<br>
why
this is the
case.<br>
It
works
sometimes, but
most of the
time when we
try to run<br>
the
test engine,
it encounters
this issue.<br>
I've
tried changing
the RCF port
by setting<br>
"TE_RCF_PORT=<some
port
number>"
and rebooting
the testbed<br>
machines.
Neither seems
to fix the
issue.<br>
<br>
It
also seems
like the
timeout takes
far longer
than 60<br>
seconds, even
when running
"export
TE_RCFUNIX_TIMEOUT=60"<br>
before
I try to run
the test
suite.<br>
I
assume the
unit for this
variable is
seconds?<br>
<br>
Thanks,<br>
Adam<br>
<br>
On
Mon, Aug 21,
2023 at
10:19 AM Adam
Hassick<br>
<<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>>
wrote:<br>
<br>
Hi
Andrew,<br>
<br>
Thanks, I've
cloned the
example
repository and
will start<br>
setting up a
configuration
for our
development
testbed<br>
today. I'll
let you know
if I run into
any
difficulties<br>
or
have any
questions.<br>
<br>
-
Adam<br>
<br>
On
Sun, Aug 20,
2023 at
4:40 AM Andrew
Rybchenko<br>
<<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a><br>
<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote:<br>
<br>
Hi Adam,<br>
<br>
I've published<br>
<a href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ts-factory/ts-rigs-sample</a><br>
<a href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/ts-rigs-sample></a>.<br>
Hopefully it will help to define your test rigs and<br>
successfully run some tests manually. Feel free to<br>
ask any questions and I'll answer here and try to<br>
update documentation.<br>
<br>
Meanwhile I'll prepare missing bits for steps (2) and<br>
(3).<br>
Hopefully everything is in place for step (4), but we<br>
need to make steps (2) and (3) first.<br>
<br>
Andrew.<br>
<br>
On 8/18/23 21:40, Andrew Rybchenko wrote:<br>
<blockquote
type="cite">
Hi Adam,<br>
<br>
> I've conferred with the rest of the team, and we<br>
think it would be best to move forward with mainly<br>
option B.<br>
<br>
OK, I'll provide the sample on Monday for you. It is<br>
almost ready right now, but I need to double-check<br>
it before publishing.<br>
<br>
Regards,<br>
Andrew.<br>
<br>
On 8/17/23 20:03, Adam Hassick wrote:<br>
<blockquote
type="cite">
Hi Andrew,<br>
<br>
I'm adding the CI mailing list to this<br>
conversation. Others in the community might find<br>
this conversation valuable.<br>
<br>
We do want to run testing on a regular basis. The<br>
Jenkins integration will be very useful for us, as<br>
most of our CI is orchestrated by Jenkins.<br>
I've conferred with the rest of the team, and we<br>
think it would be best to move forward with mainly<br>
option B.<br>
If you would like to know anything about our<br>
testbeds that would help you with creating an<br>
example ts-rigs repo, I'd be happy to answer any<br>
questions you have.<br>
<br>
We have multiple test rigs (we call these<br>
"DUT-tester pairs") that we run our existing<br>
hardware testing on, with differing network<br>
hardware and CPU architecture. I figured this might<br>
be an important detail.<br>
<br>
Thanks,<br>
Adam<br>
<br>
On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko<br>
<<a href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a><br>
<a href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote:<br>
<br>
Greatings Adam,<br>
<br>
I'm happy to hear that you're trying to bring<br>
it up.<br>
<br>
As I understand the final goal is to run it on<br>
regular basis. So, we need to make it properly<br>
from the very beginning.<br>
Bring up of all features consists of 4 steps:<br>
<br>
1. Create site-specific repository (we call it<br>
ts-rigs) which contains information about test<br>
rigs and other site-specific information like<br>
where to send mails, where to store logs etc.<br>
It is required for manual execution as well,<br>
since test rigs description is essential. I'll<br>
return to the topic below.<br>
<br>
2. Setup logs storage for automated runs.<br>
Basically it is a disk space plus apache2 web<br>
server with few CGI scripts which help a lot to<br>
save disk space.<br>
<br>
3. Setup Bublik web application which provides<br>
web interface to view testing results. Same as<br>
<a href="https://ts-factory.io/bublik"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://ts-factory.io/bublik</a><br>
<a href="https://ts-factory.io/bublik"
target="_blank" moz-do-not-send="true"><https://ts-factory.io/bublik></a><br>
<br>
4. Setup Jenkins to run tests on regularly,<br>
save logs in log storage (2) and import it to<br>
bublik (3).<br>
<br>
Last few month we spent on our homework to make<br>
it simpler to bring up automated execution<br>
using Jenkins -<br>
<a href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ts-factory/te-jenkins</a><br>
<a href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/te-jenkins></a><br>
Corresponding bits in dpdk-ethdev-ts will be<br>
available tomorrow.<br>
<br>
Let's return to the step (1).<br>
<br>
Unfortunately there is no publicly available<br>
example of the ts-rigs repository since<br>
sensitive site-specific information is located<br>
there. But I'm ready to help you to create it<br>
for UNH. I see two options here:<br>
<br>
(A) I'll ask questions and based on your<br>
answers will create the first draft with my<br>
comments.<br>
<br>
(B) I'll make a template/example ts-rigs repo,<br>
publish it and you'll create UNH ts-rigs based<br>
on it.<br>
<br>
Of course, I'll help to debug and finally bring<br>
it up in any case.<br>
<br>
(A) is a bit simpler for me and you, but (B) is<br>
a bit more generic and will help other<br>
potential users to bring it up.<br>
We can combine (A)+(B). I.e. start from (A).<br>
What do you think?<br>
<br>
Thanks,<br>
Andrew.<br>
<br>
On 8/17/23 15:18, Konstantin Ushakov wrote:<br>
<blockquote
type="cite">
Greetings
Adam,<br>
<br>
<br>
Thanks for contacting us. I copy Andrew who<br>
would be happy to help<br>
<br>
Thanks,<br>
Konstantin<br>
<br>
<blockquote
type="cite">
On 16 Aug
2023, at
21:50, Adam
Hassick<br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><ahassick@iol.unh.edu></a><br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
wrote:<br>
<br>
<br>
Greetings Konstantin,<br>
<br>
I am in the process of setting up the DPDK<br>
Poll Mode Driver test suite as an addition to<br>
our testing coverage for DPDK at the UNH lab.<br>
<br>
I have some questions about how to set the<br>
test suite arguments.<br>
<br>
I have been able to configure the Test Engine<br>
to connect to the hosts in the testbed. The<br>
RCF, Configurator, and Tester all begin to<br>
run, however the prelude of the test suite<br>
fails to run.<br>
<br>
<a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters</a>
<a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true"><https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters></a><br>
<br>
The documentation mentions that there are<br>
several test parameters for the test suite,<br>
like for the IUT test link MAC, etc. These<br>
seem like they would need to be set somewhere<br>
to run many of the tests.<br>
<br>
I see in the Test Engine documentation, there<br>
are instructions on how to create new<br>
parameters for test suites in the Tester<br>
configuration, but there is nothing in the<br>
user guide or in the Tester guide for how to<br>
set the arguments for the parameters when<br>
running the test suite that I can find. I'm<br>
not sure if I need to write my own Tester<br>
config, or if I should be setting these in<br>
some other way.<br>
<br>
How should these values be set?<br>
<br>
I'm also not sure what environment<br>
variables/arguments are strictly necessary or<br>
which are optional.<br>
<br>
Regards,<br>
Adam<br>
<br>
-- *Adam Hassick*<br>
Senior Developer<br>
UNH InterOperability Lab<br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a href="http://iol.unh.edu" target="_blank"
moz-do-not-send="true">iol.unh.edu</a> <a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1 (603) 475-8248<br>
</blockquote>
</blockquote>
<br>
<br>
<br>
-- *Adam Hassick*<br>
Senior Developer<br>
UNH InterOperability Lab<br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a href="http://iol.unh.edu" target="_blank"
moz-do-not-send="true">iol.unh.edu</a> <a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1 (603) 475-8248<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
<br>
--
*Adam Hassick*<br>
Senior
Developer<br>
UNH
InterOperability
Lab<br>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1
(603) 475-8248<br>
<br>
<br>
<br>
--
*Adam
Hassick*<br>
Senior
Developer<br>
UNH
InterOperability
Lab<br>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1
(603) 475-8248<br>
</blockquote>
<br>
<br>
<br>
--
*Adam Hassick*<br>
Senior
Developer<br>
UNH
InterOperability
Lab<br>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1 (603)
475-8248<br>
</blockquote>
<br>
<br>
<br>
-- <br>
*Adam Hassick*<br>
Senior
Developer<br>
UNH
InterOperability
Lab<br>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1 (603)
475-8248<br>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br
clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br
clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br
clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
<div style="white-space:normal"> </div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam
Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>