<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hi Adam,<br>
<br>
fresh ts-rigs-sample has all required bindings to publish logs to
ts-factory.io. See the top-most commit.<br>
So, you just need tune SSH to use correct user, port and key for
ts-factory.io. You can add below settings<br>
to corresponding user .ssh/config or add corresponding options to
sftp command in scripts/publish_logs/prj/ts-factory/publish:<br>
<br>
Host ts-factory.io<br>
User unh-iol<br>
Port 56777<br>
IdentitiesOnly yes<br>
IdentityFile <path-to-private-key><br>
<br>
The publishing is asynchronous in ts-factory case. i.e. user puts
logs and cron job checks incoming directory every 10 minutes and
publish found logs.<br>
<br>
You can request publishing when you start testing using --publish
option or publish previous run testing results using
./scripts/publish_logs from dpdk-ethdev-ts sources. PWD should be
the directory you run tests from.<br>
<br>
Please, let me know how it goes.<br>
<br>
Regards,<br>
Andrew.<br>
<br>
On 10/26/23 20:44, Adam Hassick wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAC-YWqiQJUJMc6GjHbkEz8=+Mvnk5XEq=v3g8Kh-iVHxgHkwCA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
I've attached the public key we'd like to use.<br>
<br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Oct 26, 2023 at
8:19 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
On 10/25/23 23:27, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Andrew,<br>
<br>
</div>
<div>Sorry about the two week radio silence, we're still
trying to sort out logistics for deployment on our
end.<br>
</div>
<div><br>
</div>
<div>> I've created setup on <a
href="http://ts-factory.io" target="_blank"
moz-do-not-send="true">ts-factory.io</a> which
allows to publish logs. Let me know if you'd like to
try it and I'll provide credentials, script and short
instruction.</div>
<div><br>
</div>
<div>We're interested in publishing some test logs to
the ts-factory Bublik instance in the meantime.<br>
</div>
</div>
</blockquote>
<br>
Please, send me SSH public key which you'd like to use to
upload logs.<br>
I'll provide helper script and instructions to do it.<br>
<br>
Andrew.<br>
<blockquote type="cite">
<div dir="ltr"><br>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Oct 23, 2023
at 7:11 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
> Now that our test results are in line with
yours, we can begin looking into setting up the
production environment.<br>
<br>
Please, let me know if you need any help with it
or waiting for an input from me.<br>
<br>
Regards,<br>
Andrew.<br>
<br>
On 10/10/23 17:09, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
Thank you for taking a look at our
log. Netplan was attempting to run
DHCP on our test links, and
additionally I discovered that the
NIC firmware was transmitting LLDP
packets, causing tests to fail in
the same way. Now that these
problems have been fixed, our pass
rate on the XL710 is approximately
91%. Now that our test results are
in line with yours, we can begin
looking into setting up the
production environment.</div>
</div>
<br>
</div>
First, is it possible to run the test
agent on ARM hosts? Our ARM testbeds have
the best topology for running this test
suite, with separate tester and DUT
servers.<br>
<br>
</div>
We are testing this test suite on two x86
development servers using the test suite's
recommended server topology. In contrast,
our existing x86 production testbeds which
run DTS have a single server topology. This
single server has both the tester NIC and
the device under test NIC installed, with
NUMA node separation between TRex and DPDK.
We're going to test running the two test
agent processes on the single-server
testbeds if we cannot run this on ARM. Is
there any reason you can think of that would
prevent this setup from working?<br>
<br>
</div>
<div>Once we figure out where this can live in
production, then we will begin setting up
log storage, Jenkins integration, and
Bublik.<br>
</div>
<div><br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Oct 5,
2023 at 6:25 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
> Do these default to vfio-pci?<br>
<br>
Yes, vfio-pci is the default.<br>
However, it does not work in the case of
Mellanox which uses bifurcated driver. It
should mlx5_core for Mellanox NICs.<br>
<br>
> Here is the text log from a run on
our Intel XL710 NICs, with the expected
result profile set to the X710.<br>
<br>
It is hard to analyze all tests using text
logs, but I definitely see one common
problem. Tests receive unexpected packets
and fail because of it.<br>
Tests are written very strict from this
point of view and it brought fruits in the
past when HW had bugs.<br>
Are DUT and tester connected back-to-back
on tested interfaces or via switch?<br>
If via switch, is it possible to isolate
it from everything else?<br>
If back-to-back, it could be some embedded
SW/FW which regenerates these packet.<br>
I definitely see unexpected DHCP packets.<br>
<br>
> We haven't set up the Jenkins
integration yet, however if this is
required to import the logs then we will
prioritize that.<br>
<br>
Unfortunately manual runs do not generate
all artifacts required to import logs.
However, we have almost solved it right
now. Hopefully we'll finalize it in a day
or two. I'll let you know when these
changes are available.<br>
<br>
Regards,<br>
Andrew.<br>
<br>
On 10/4/23 16:48, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
Ok, that makes sense. I don't see
TE_ENV_H1/H2_DPDK_DRIVER set
anywhere in the default
configurations for the Intel X710.
Do these default to vfio-pci?<br>
</div>
<div>We have IOMMU enabled on our
development testbed, and should be
able to bind vfio-pci.<br>
</div>
<div>Here is the text log from a run
on our Intel XL710 NICs, with the
expected result profile set to the
X710. We haven't set up the
Jenkins integration yet, however
if this is required to import the
logs then we will prioritize that.<br>
</div>
<div
class="gmail_chip gmail_drive_chip"
style="width:396px;height:18px;max-height:18px;background-color:rgb(245,245,245);padding:5px;color:rgb(34,34,34);font-family:arial;font-style:normal;font-weight:bold;font-size:13px;border:1px solid rgb(221,221,221);line-height:1"><a
href="https://drive.google.com/file/d/10N5JfxFMP7lNXDBgJeN_z-NL2JNUY7nJ/view?usp=drive_web"
style="display:inline-block;max-width:366px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;text-decoration:none;padding:1px 0px;border:medium"
aria-label="log.txt.tar.gz"
target="_blank"
moz-do-not-send="true"><img
style="vertical-align: bottom; border: medium;"
src="https://ssl.gstatic.com/docs/doclist/images/icon_10_generic_list.png"
moz-do-not-send="true"> <span
dir="ltr"
style="color:rgb(17,85,204);text-decoration:none;vertical-align:bottom">log.txt.tar.gz</span></a><img
style="opacity: 0.55; float: right; display: none;"
moz-do-not-send="true"></div>
<br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Mon, Sep 18, 2023 at 11:04 AM Andrew
Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>On 9/18/23 17:44, Adam Hassick
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>Hi Andrew and
Konstantin,<br>
<br>
</div>
Thank you for adding
the tester-dial
feature, this opens
up the possibility
for us to do CI
integrated testing
in the future.<br>
<br>
</div>
Our Mellanox pass rate
is similar to yours
(about ~2400 passing,
~4400 failing),
however our Intel pass
rates are far worse.<br>
</div>
<div>I will try running
tests on the XL710
with the trc-tags
argument set and see
if it improves the
pass rate.<br>
</div>
Another thing I noticed
in the results you
uploaded is that the
results are tagged with
vfio-pci and not i40e.</div>
<div>Though in the
environment dump, the
driver on the test
machine and the DUT are
set to use the i40e
driver. Is this
important at all?<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
I think it is a misunderstanding
here. There are two kinds of driver
in configuration: net driver and
so-called DPDK driver.<br>
Net driver is a Linux kernel network
device driver used on Tester side.<br>
DPDK driver is a Linux kernel driver
to bind device to to use it with
DPDK. So, it is NOT a driver inside
DPDK (drivers/net/*).<br>
In the case of bifurcated driver
(like mlx5_core) it is the same in
both cases.<br>
In non-bifurcated case DPDK driver
is some UIO driver(vfio-pci,
uio-pci-generic or igb_uio).<br>
Some expectations depend on used
UIO. For example, uio-pci-generic do
not support many interrupts (used by
usecases/rx_intr test cases).<br>
That's why we care corresponding TRC
tag.<br>
<br>
TE_ENV_*_DPDK_DRIVER variables
should be vfio-pc in 710's Intel
case. Or uio-pci-generic if IOMMU is
turned off on corresponding machines
and Linux distro does not support
VFIO no IOMMU mode.<br>
<br>
Andrew.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>There isn't anything
preventing us from pushing
our results up to the
existing Bublik instance
running at <a
href="http://ts-factory.io" target="_blank" moz-do-not-send="true">ts-factory.io</a>
that I can think of at the
moment.<br>
</div>
<div>We will have to work
out how to submit our
results to your Bublik
instance in a controlled
and secure manner in that
case.<br>
</div>
<div>As far as I know we
won't need access controls
for the results
themselves. I'll discuss
this with Patrick and will
let you know once we
confirm that it's fine.</div>
</div>
<div><br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Mon, Sep
18, 2023 at 2:26 AM Andrew
Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>On 9/18/23 09:23,
Konstantin Ushakov wrote:<br>
</div>
<blockquote type="cite">
<div
style="font-family:sans-serif">
<div
style="white-space:normal">
<p dir="auto">Hi
Andrew,</p>
<p dir="auto">should
we always
auto-assign the tags
or you don’t do it
since it slows down
(by some seconds)
the TE startup?</p>
</div>
</div>
</blockquote>
<br>
Tags are auto-assigned, but
I guess it differs in Adam's
case since NIC is a bit
different. Below test will
help to understand if it is
the root cause of very
different expectations. If
pass rate will be close to
mine, I'll simply update TRC
database to share
expectations for mine NIC
and NIC used by Adam.<br>
<br>
<blockquote type="cite">
<div
style="font-family:sans-serif">
<div
style="white-space:normal">
<p dir="auto">Hi Adam,</p>
<p dir="auto">I think
I second the
question from Andrew
- happy to help you
with the triage so
that we get to the
same baseline. Do
you have a good way
for us to share the
logs? I.e. say
upload to ts-factory
if we add strict
permissions system
so it’s not
publishing or any
other way.</p>
<p dir="auto">Thanks,
<br>
Konstantin</p>
<br>
<p dir="auto">On 18
Sep 2023, at 9:15,
Andrew Rybchenko
wrote:</p>
</div>
<blockquote
style="margin:0px 0px 5px;padding-left:5px;border-left:2px solid rgb(119,119,119);color:rgb(119,119,119)">
<div
id="m_-8945420805490880521m_1676397175446446606m_3634070300823225067m_-4480488732519633353m_-792332968217640304733A56D0A-0ED3-47F6-99B4-35C92E41C2DA">
<div>Hi Adam,<br>
<br>
I've uploaded
fresh testing
results to <a
href="http://ts-factory.io" target="_blank" moz-do-not-send="true">ts-factory.io</a>
[1] to be on the
same page.<br>
<br>
I think I know why
your and mine
results on Intel
710 series NICs
differ so much.
Testing results
expectations
database
(dpdk-ethdev-ts/trc/*)
is filled in in
terms of TRC
tags. I.e.
expectations
depends on TRC
tags discovered by
helper scripts
when testing is
started. These
tags identify
various aspects of
what is tested.
Ideally
expectations
should be written
in terms of root
cause of the
expected
behaviour. If it
is a driver
expectations,
driver tag should
be used. If it is
HW limitation,
tags with PCI IDs
should be used.
However, it is not
always easy to
classify it
correctly if
you're not
involved in driver
development. So,
in order case
expectations for
710's Intel are
filled in in terms
of PCI IDs. I
guess PCI ID
differ in your
case and that's
why expectations
filled in for my
NIC do not apply
to your runs.<br>
<br>
Just try to add
the following
option when you
run on your 710's
Intel in order to
mimic mine and see
if it helps to
achieve better
pass rate.<br>
--trc-tag=pci-8086-1572<br>
<br>
BTW, fresh TE tag
<span>v1.21.0 has
improved
algorithm to
choose tests for
--tester-dial
option. It
should have
better coverage
now.</span><br>
<br>
Andrew.<br>
<br>
[1] <a
href="https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1"
target="_blank"
moz-do-not-send="true">https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1</a><br>
<br>
On 9/13/23 18:45,
Andrew Rybchenko
wrote:<br>
</div>
<blockquote
type="cite">
<div>Hi Adam,<br>
<br>
I've pushed new
TE tag v1.20.0
which supported
a new
command-line
option
--tester-dial=NUM
where NUM is
from 0 to 100.
it allows to
choose
percentage of
tests to run. If
you want stable
set, you should
pass
--tester-random-seed=0
(or other
integer). It is
the first sketch
and we have
plans to improve
it, but feedback
would be
welcome.<br>
<br>
> Is it
needed on the
tester?<br>
<br>
It is hard to
say if it is
strictly
required for
simple tests.
However, it is
better to update
Tester as well,
since
performance
tests run DPDK
on Tester as
well.<br>
<br>
> Are there
any other manual
setup steps for
these devices
that I might be
missing?<br>
<br>
I don't remember
anything else.<br>
<br>
I think it is
better to get
down to details
and take a look
at logs. I'm
ready to help
with it and
explain what's
happening there.
May be it will
help to
understand if it
is a problem
with
setup/configuration.<br>
<br>
Text logs are
not very
convenient.
Ideally logs
should be
imported to
bublik, however,
manual runs do
not provide all
required
artifacts right
now (Jenkins
jobs generate
all required
artifacts).<br>
Other option is
'tmp_raw_log'
file (should be
packed to make
it smaller)
which could be
converted to
various log
formats.<br>
Would it be OK
for you if I
import your logs
to bublik at <a
href="http://ts-factory.io" target="_blank" moz-do-not-send="true">ts-factory.io</a>?
Or is it a
problem that it
is publicly
available?<br>
Would it help if
we add
authentication
and access
control there?<br>
<br>
Andrew.<br>
<br>
On 9/8/23 17:57,
Adam Hassick
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi
Andrew,<br>
<br>
</div>
I have a
couple
questions
about needed
setup of the
NICs for the
ethdev test
suite.<br>
<br>
</div>
Our MCX5s and
XL710s are
failing the
checkup tests.
The pass rate
appears to be
much worse on
the XL710s (40
of 73 tests
failed, 3
passed
unexpectedly).<br>
<br>
</div>
For the
XL710s, I've
updated the
driver and NVM
versions to
match the
minimum
supported
versions in
the
compatibility
matrix found
on the DPDK
documentation.
This did not
change the
failure rate
much.<br>
</div>
For the MCX5s,
I've installed
the latest LTS
version of the
OFED
bifurcated
driver on the
DUT. Is it
needed on the
tester?<br>
<br>
</div>
Are there any
other manual
setup steps
for these
devices that I
might be
missing?<br>
<div>
<div>
<div>
<div>
<div>
<div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Sep 6, 2023 at 11:00 AM Adam Hassick <<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi
Andrew,<br>
<br>
</div>
<div>Yes, I
copied the
X710 configs
to set up
XL710 configs.
I changed the
environment
variable names
from the X710
suffix to
XL710 suffix
in the script,
and forgot to
change them in
the
corresponding
environment
file.<br>
</div>
</div>
That fixed the
issue.<br>
<br>
</div>
I got the
checkup tests
working on the
XL710 now.
Most of them
are failing,
which leads me
to believe
this is an
issue with our
testbed. Based
on the DPDK
documentation
for i40e, the
firmware and
driver
versions are
much older
than what DPDK
22.11 LTS and
main prefer,
so I'll try
updating
those.<br>
<br>
</div>
For now I'm
working on
getting the
XL710 checkup
tests passing,
and will pick
up getting the
E810
configured
properly next.
I'll let you
know if I run
into any more
issues in
relation to
the test
engine.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
On 9/5/23
18:01, Adam
Hassick wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi
Andrew,<br>
<br>
</div>
The
compilation
warning issue
is now
resolved.
Again, thank
you guys for
fixing this
for us. I can
run the tests
on the
Mellanox CX5s
again, however
I'm running
into a couple
new issues
with running
the prologues
on the Intel
cards.<br>
<br>
</div>
When running
testing on the
Intel XL710s,
I see this
error appear
in the log:<br>
<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">ERROR
prologue
Environment
LIB
14:16:13.650<br>
Too few
networks in
available
configuration
(0) in
comparison
with required
(1)<br>
</blockquote>
<br>
</div>
This seems
like a trivial
configuration
error, perhaps
this is
something I
need to set up
in ts-rigs. I
briefly
searched
through the
examples there
and didn't see
any mention of
how to set up
a network.<br>
</div>
<div>I will
attach this
log just in
case you need
more
information.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Unfortunately
logs are
insufficient
to understand
it. I've
pushed new tag
to TE v1.19.0
which add log
message with
TE_*
environment
variables.<br>
Most likely
something is
wrong with
variables
which are used
as conditions
when available
networks are
defined in
ts-conf/cs/inc.net_cfg_pci_fns.yml:<br>
TE_PCI_INSTANCE_IUT_TST1<br>
TE_PCI_INSTANCE_IUT_TST1a<br>
TE_PCI_INSTANCE_TST1a_IUT<br>
TE_PCI_INSTANCE_TST1_IUT<br>
My guess it
that you
change naming
a bit, but
script like
ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>There is
a different
error when
running on the
Intel E810s.
It appears to
me like it
starts DPDK,
does some
configuration
inside DPDK
and on the
device, and
then fails to
bring the
device back
up. Since this
error seems
very
non-trivial, I
will also
attach this
log.<br>
</div>
</div>
</div>
</blockquote>
<br>
This one is a
bit simpler.
Few lines
after the
first ERROR in
log I see the
following:<br>
WARN RCF
DPDK
13:06:00.144<br>
ice_program_hw_rx_queue(): currently package doesn't support RXDID (22)<br>
ice_rx_queue_start(): fail to program RX queue 0<br>
ice_dev_start(): fail to start Rx queue 0<br>
Device with
port_id=0
already
stopped<br>
<br>
It is
stdout/stderr
from test
agent which
runs DPDK.
Same logs in
plain format
are available
in ta.DPDK
file.<br>
I'm not an
expert here,
but I vaguely
remember that
E810 requires
correct
firmware and
DDP to be
loaded.<br>
There is some
information in
dpdk/doc/guides/nics/ice.rst.<br>
<br>
You can try to
add
--dev-args=safe-mode-support=1
command-line
option
described
there.<br>
<br>
Hope it helps,<br>
Andrew.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div><br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Fri, Sep 1, 2023 at 3:59 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
On 8/31/23
22:38, Adam
Hassick wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>Hi
Andrew,<br>
</div>
<div><br>
I have one
additional
question as
well: Does the
test engine
support
running tests
on two ARMv8
test agents?</div>
<div><br>
</div>
<div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">1.
We'll sort out
warnings this
week. Thanks
for heads up.<br>
</blockquote>
<div><br>
</div>
<div>Great.
Let me know
when that's
fixed.</div>
</div>
</div>
</blockquote>
<br>
Done. We also
fixed a number
of warnings in
TE.<br>
Also we fixed
root test
package name
to be
consistent
with the
repository
name.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>Support
for old LTS
branches was
dropped some
time ago, but
in the future
it is
definitely
possible to
keep it for
new LTS
branches. I
think 22.11 is
supported, but
I'm not sure
about older
LTS releases.</div>
</blockquote>
<div><br>
</div>
<div>Good to
know.<br>
<div> <br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2.
You can add
command-line
option
--sanity to
run tests
marked with
TEST_HARNESS_SANITY
requirement
(see
dpdk-ethdev-ts/scripts/run.sh
and grep
TEST_HARNESS_SANITY
dpdk-ethdev-ts
to see which
tests are
marked). Yes,
there is a
space for
terminology
improvement
here. We'll do
it.<br>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
Done. Now it
is called
--checkup.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Also it takes
a lot of time
because of
failures and
tests which
wait for some
timeout.<br>
</blockquote>
</div>
<div><br>
</div>
<div>That
makes sense to
me. We'll use
the time to
complete tests
on virtio or
the Intel
devices as a
reference for
how long the
tests really
take to
complete.<br>
</div>
<div>We will
explore the
possibility of
periodically
running the
sanity tests
for patches.<br>
</div>
</div>
</div>
</blockquote>
<br>
I'll
double-check
and let you
know how long
entire TS runs
on Intel X710,
E810, Mellanox
CX5 and virtio
net. Just to
ensure that
time observed
in your case
looks the
same.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div> <br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The
test harness
can provide
coverage
reports based
on gcov, but
I'm not sure
what you mean
by a "dial" to
control test
coverage.
Provided
reports are
rather for
human to
analyze.<br>
</blockquote>
</div>
<div><br>
</div>
<div>The
general idea
is to have
some kind of
parameter on
the test
suite, which
could be an
integer
ranging from
zero to ten,
that controls
how many tests
are run based
on how
important the
test is.<br>
<br>
</div>
<div>Similar
to how some
command line
interfaces
provide a
verbosity
level
parameter
(some number
of "-v"
arguments) to
control the
importance of
the
information in
the log.<br>
</div>
The verbosity
level zero
only prints
very important
log messages,
while ten
prints
everything.<br>
</div>
<div><br>
In much the
same manner as
above, this
"dial"
parameter
controls what
tests are run
and with what
parameters
based on how
important
those tests
and test
parameter
combinations
are.<br>
Coverage Level
zero tells the
suite to run a
very basic set
of important
tests, with
minimal
parameterization.
This mode
would take
only ~5-10
minutes to
run.<br>
In contrast,
Coverage Level
ten includes
all the edge
cases, every
combination of
test
parameters,
everything the
test suite can
do, which
takes the
normal several
hours to run.<br>
The values 1 -
9 are between
those two
extremes,
allowing the
user to get a
gradient of
test coverage
in the results
and to limit
the running
time.<br>
<br>
</div>
Then we could,
for example,
run the
"run.sh" with
a level of 2
or 3 for
incoming
patches that
need quick
results, and
with a level
of 10 for the
less often run
periodic tests
performed on
main or LTS
branches.<br>
</div>
</blockquote>
<br>
Understood
now. Thanks a
lot for the
idea. We'll
discuss it and
come back.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>3. Yes,
really many
tests on
Mellanox CX5
NICs report
unexpected
testing
results.
Unfortunately
it is time
consuming to
fill in
expectations
database since
it is
necessary to
analyze
testing
results and
classify if it
is a bug or
just
acceptable
behaviour
aspect.<br>
<br>
Bublik allows
to compare
results of two
runs. It is
useful for
human, but
still not good
for
automation.<br>
<br>
I have local
patch for mlx5
driver which
reports Tx
ring size
maximum. It
makes pass
rate higher.
It is a
problem for
test harness
that mlx5 does
not report
limits right
now.<br>
<br>
Pass rate on
Intel X710 is
about 92% on
my test rig.
Pass rate on
virtio net is
99% right now
and could be
done 100%
easily (just
one thing to
fix in
expectations).<br>
<br>
I think logs
storage setup
is essential
for logs
analysis. Of
course, you
can request
HTML logs when
you run tests
(--log-html=html) or generate after run using
dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a browser, but
logs storage
makes it more
convenient.<br>
</div>
</div>
</blockquote>
<div><br>
We are
interested in
setting up
Bublik,
potentially as
an
externally-facing
component,
once we have
our process of
running the
test suite
stabilized.</div>
<div>Once we
are able to
run the test
suite again,
I'll see what
the pass rate
is on our
other
hardware.<br>
Good to know
that it isn't
an issue with
our dev
testbed
causing the
high fail
rate.</div>
</div>
<div>
<div><br>
</div>
<div>For Intel
hardware, we
have an XL710
and an Intel
E810-C in our
development
testbed.
Although they
are slightly
different
devices,
ideally the
pass rate will
be identical
or similar. I
have yet to
set up a VM
pair for
virtio, but we
will soon.<br>
</div>
<div><br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Latest
version of
test-environment
has examples
of our CGI
scripts which
we use for log
storage (see
tools/log_server/README.md).<br>
<br>
Also all bits
for Jenkins
setup are
available. See
dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
ts-rigs-sample.<br>
</blockquote>
</div>
<div><br>
</div>
<div>Jenkins
integration,
setting up
production rig
configurations, and permanent log storage will be our next steps once I
am able to run
the tests
again.<br>
</div>
<div>Unless
there is an
easy way to
have meson not
pass "-Werror"
into GCC. Then
I would be
able to run
the test
suite.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Hopefully it
is resolved
now.<br>
<br>
I thought a
bit more about
your usecase
for Jenkins.
I'm not 100%
sure that
existing
pipelines are
convenient for
your usecase.<br>
Fill free to
ask questions
when you are
on it.<br>
<br>
Thanks,<br>
Andrew.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
<div><br>
</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div><br>
On 8/29/23
17:02, Adam
Hassick wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi
Andrew,<br>
<br>
</div>
That fix seems
to have
resolved the
issue, thanks
for the quick
turnaround
time on that
patch.<br>
</div>
<div>Now that
we have the
RCF timeout
issue
resolved,
there are a
few other
questions and
issues that we
have about the
tests
themselves.</div>
<br>
</div>
<div>1. The
test suite
fails to build
with a couple
warnings.<br>
</div>
<div><br>
</div>
<div>Below is
the stderr log
from
compilation:<br>
</div>
<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">FAILED:
<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
cc
-Ilib/76b5a35@@ts_dpdk_pmd@sta
-Ilib
-I../../lib
-I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
-fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
-Winvalid-pch
-Werror -g
-D_GNU_SOURCE
-O0 -ggdb
-Wall -W -fPIC
-MD -MQ '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-MF '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
-o '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-c
../../lib/dpdk_pmd_ts.c<br>
../../lib/dpdk_pmd_ts.c: In function
‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
no format
arguments
[-Werror=format-security]<br>
5577 | rc
=
te_kvpair_add(result,
buf, mode);<br>
| ^~<br>
cc1: all
warnings being
treated as
errors<br>
ninja: build
stopped:
subcommand
failed.<br>
ninja:
Entering
directory `.'<br>
FAILED: <a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
cc
-Ilib/76b5a35@@ts_dpdk_pmd@sta
-Ilib
-I../../lib
-I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
-fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall
-Winvalid-pch
-Werror -g
-D_GNU_SOURCE
-O0 -ggdb
-Wall -W -fPIC
-MD -MQ '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-MF '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
-o '<a
href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-c
../../lib/dpdk_pmd_ts.c<br>
../../lib/dpdk_pmd_ts.c: In function
‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
no format
arguments
[-Werror=format-security]<br>
5577 | rc
=
te_kvpair_add(result,
buf, mode);<br>
| ^~<br>
cc1: all
warnings being
treated as
errors<br>
</blockquote>
<div>
<div>
<div><br>
</div>
<div>This
error wasn't
occurring last
week, which
was the last
time I ran the
tests.<br>
</div>
<div>The TE
host and the
DUT have GCC
v9.4.0
installed, and
the tester has
GCC v11.4.0
installed, if
this
information is
helpful.<br>
</div>
<div><br>
</div>
<div>2. On the
Mellanox CX5s,
there are over
6,000 tests
run, which
collectively
take around 9
hours. Is it
possible, and
would it make
sense, to
lower the test
coverage and
have the test
suite run
faster?<br>
<br>
</div>
<div>For some
context, we
run immediate
testing on
incoming
patches for
DPDK main and
development
branches, as
well as
periodic test
runs on the
main, stable,
and LTS
branches.<br>
</div>
<div>For us to
consider
including this
test suite as
part of our
immediate
testing on
patches, we
would have to
reduce the
test coverage
to the most
important
tests.<br>
This is
primarily to
reduce the
testing time
to, for
example, less
than 30
minutes.
Testing on
patches can't
take too long
because the
lab can
receive
numerous
patches each
day, which
each require
individual
testing runs.<br>
<br>
</div>
<div>At what
frequency we
run these
tests, and on
what, still
needs to be
discussed with
the DPDK
community, but
it would be
nice to know
if the test
suite had a
"dial" to
control the
testing
coverage.<br>
</div>
<div><br>
</div>
<div>3. We see
a lot of test
failures on
our Mellanox
CX5 NICs.
Around 2,300
of ~6,600
tests passed.
Is there
anything we
can do to
diagnose these
test failures?<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Tue, Aug 29, 2023 at 8:07 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
I've pushed
the fix in
main branch
and a new tag
v1.18.1. It
should solve
the problem
with IPv6
address from
DNS.<br>
<br>
Andrew.<br>
<br>
On 8/29/23
00:05, Andrew
Rybchenko
wrote:<br>
</div>
<blockquote
type="cite">
<div>Hi Adam,<br>
<br>
> Does the
test engine
prefer to use
IPv6 over IPv4
for initiating
the RCF
connection to
the test bed
hosts? And if
so, is there a
way to force
it to use
IPv4?<br>
<br>
Brilliant
idea. If DNS
returns both
IPv4 and IPv6
addresses in
your case, I
guess it is
the root cause
of the
problem.<br>
Of course, it
is TE problem
since I see
really weird
code in
lib/comm_net_engine/comm_net_engine.c
line 135.<br>
<br>
I've pushed
fix to the
branch
user/arybchik/fix_ipv4_only
in
ts-factory/test-environment
repository.
Please, try.<br>
<br>
It is late
night fix with
minimal
testing and no
review. I'll
pass it
through review
process
tomorrow and<br>
hopefully it
will be
released in
one-two days.<br>
<br>
Andrew.<br>
<br>
On 8/28/23
18:02, Adam
Hassick wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi
Andrew,<br>
<br>
</div>
We have yet to
notice a
distinct
pattern with
the failures.
Sometimes, the
RCF will start
and connect
without issue
a few times in
a row before
failing to
connect again.
Once the issue
begins to
occur, neither
rebooting all
of the hosts
(test engine
VM, tester,
IUT) or
deleting all
of the build
directories
(suites,
agents, inst)
and rebooting
the hosts
afterward
resolves the
issue. When it
begins working
again seems
very arbitrary
to us.<br>
<br>
</div>
<div>I do
usually try to
terminate the
test engine
with Ctrl+C,
but when it
hangs while
trying to
start RCF,
that does not
work.<br>
</div>
<div><br>
</div>
<div>Does the
test engine
prefer to use
IPv6 over IPv4
for initiating
the RCF
connection to
the test bed
hosts? And if
so, is there a
way to force
it to use
IPv4?<br>
<br>
</div>
<div> - Adam<br>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>> I'll
double-check
test engine on
Ubuntu 20.04
and Ubuntu
22.04.<br>
<br>
Done. It works
fine for me
without any
issues.<br>
<br>
Have you
noticed any
pattern when
it works or
does not work?<br>
May be it is a
problem of not
clean state
after
termination?<br>
Does it work
fine the first
time after
DUTs reboot?<br>
How do you
terminate
testing? It
should be done
using Ctrl+C
in terminal
where you
execute run.sh
command.<br>
In this case
it should
shutdown
gracefully and
close all test
agents and
engine
applications.<br>
<br>
(I'm trying to
understand why
you've seen
many test
agent
processes. It
should not
happen.)<br>
<br>
Andrew.<br>
<br>
On 8/25/23
17:41, Andrew
Rybchenko
wrote:<br>
</div>
<blockquote
type="cite">
<div>On
8/25/23 17:06,
Adam Hassick
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>Hi
Andrew,<br>
<br>
</div>
Two of our
systems (the
Test Engine
runner and the
DUT host) are
running Ubuntu
20.04 LTS,
however this
morning I
noticed that
the tester
system (the
one having
issues) is
running Ubuntu
22.04 LTS.<br>
</div>
<div>This
could be the
source of the
problem. I
encountered a
dependency
issue trying
to run the
Test Engine on
22.04 LTS, so
I downgraded
the system.
Since the
tester is also
the host
having
connection
issues, I will
try
downgrading
that system to
20.04, and see
if that
changes
anything.<br>
</div>
</div>
</blockquote>
<br>
Unlikely, but
who knows. We
run tests
(DUTs) on
Ubuntu 20.04,
Ubuntu 22.04,
Ubuntu 22.10,
Ubuntu 23.04,
Debian 11 and
Fedora 38
every night.<br>
Right now
Debian 11 is
used for test
engine in
nightly
regressions.<br>
<br>
I'll
double-check
test engine on
Ubuntu 20.04
and Ubuntu
22.04.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div>I did try
passing in the
"--vg-rcf"
argument to
the run.sh
script of the
test suite
after
installing
valgrind, but
there was no
additional
output that I
saw.<br>
</div>
</div>
</blockquote>
<br>
Sorry, I
should
valgrind
output should
be in
valgrind.te_rcf
(direction
where you run
test engine).<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I will
try pulling in
the changes
you've pushed
up, and will
see if that
fixes
anything.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hello
Adam,<br>
<br>
On 8/24/23
23:54, Andrew
Rybchenko
wrote:<br>
</div>
<blockquote
type="cite">I'd
like to try to
repeat the
problem
locally. Which
Linux distro
is running on
test engine
and agents?<br>
<br>
In fact I know
one problem
with Debian 12
and Fedora 38
and we have<br>
patch in
review to fix
it, however,
the behaviour
is different
in<br>
this case, so
it is unlike
the same
problem.<br>
</blockquote>
<br>
I've just
published a
new tag which
fixes known
test engine
side problems
on Debian 12
and Fedora 38.<br>
<br>
<blockquote
type="cite"><br>
One more idea
is to install
valgrind on
the test
engine host
and<br>
run with
option
--vg-rcf to
check if
something
weird is
happening.<br>
<br>
What I don't
understand
right now is
why I see just
one failed
attempt<br>
to connect in
your log.txt
and then
Logger
shutdown after
9 minutes.<br>
<br>
Andrew.<br>
<br>
On 8/24/23
23:29, Adam
Hassick wrote:<br>
<blockquote
type="cite"> >
Is there any
firewall in
the network or
on test hosts
which could
block incoming
TCP connection
to the port
23571 <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host
where you run
test engine?<br>
<br>
Our test
engine host
and the
testbed are on
the same
subnet. The
connection
does work
sometimes.<br>
<br>
> If
behaviour the
same on the
next try and
you see that
test agent is
kept running,
could you
check using<br>
><br>
> #
netstat -tnlp<br>
><br>
> that
Test Agent is
listening on
the port and
try to
establish TCP
connection
from test
agent using<br>
><br>
> $ telnet
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank"
moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a> <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571 <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a><br>
><br>
> and
check if TCP
connection
could be
established.<br>
<br>
I was able to
replicate the
same behavior
again, where
it hangs while
RCF is trying
to start.<br>
Running this
command, I see
this in the
output:<br>
<br>
tcp 0
0 <a
href="http://0.0.0.0:23571" target="_blank" moz-do-not-send="true">0.0.0.0:23571</a>
<a
href="http://0.0.0.0:23571" target="_blank" moz-do-not-send="true"><http://0.0.0.0:23571></a>
0.0.0.0:*
LISTEN
18599/ta<br>
<br>
So it seems
like it is
listening on
the correct
port.<br>
Additionally,
I was able to
connect to the
Tester machine
from our Test
Engine host
using telnet.
It printed the
PID of the
process once
the connection
was opened.<br>
<br>
I tried
running the
"ta"
application
manually on
the command
line, and it
didn't print
anything at
all.<br>
Maybe the
issue is
something on
the Test
Engine side.<br>
<br>
On Thu, Aug
24, 2023 at
2:35 PM Andrew
Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote:<br>
<br>
Hi Adam,<br>
<br>
> On
the tester
host (which
appears to be
the Peer
agent), there<br>
are four
processes that
I see running,
which look
like the test<br>
agent
processes.<br>
<br>
Before the
next try I'd
recommend to
kill these
processes.<br>
<br>
Is there
any firewall
in the network
or on test
hosts which
could<br>
block
incoming TCP
connection to
the port 23571<br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host<br>
where you
run test
engine?<br>
<br>
If
behaviour the
same on the
next try and
you see that
test agent is<br>
kept
running, could
you check
using<br>
<br>
# netstat
-tnlp<br>
<br>
that Test
Agent is
listening on
the port and
try to
establish TCP<br>
connection
from test
agent using<br>
<br>
$ telnet <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank"
moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a><br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571<br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a><br>
<br>
and check
if TCP
connection
could be
established.<br>
<br>
Another
idea is to
login Tester
under root as
testing does,
get<br>
start TA
command from
the log and
try it by
hands without
-n and<br>
remove
extra
escaping.<br>
<br>
# sudo
PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1<br>
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=<br>
<br>
Hopefully
in this case
test agent
directory
remains in the
/tmp and<br>
you don't
need to copy
it as testing
does.<br>
May be
output could
shed some
light on
what's going
on.<br>
<br>
Andrew.<br>
<br>
On 8/24/23
17:30, Adam
Hassick wrote:<br>
<blockquote
type="cite">
Hi Andrew,<br>
<br>
This is
the output
that I see in
the terminal
when this
failure<br>
occurs,
after the test
agent binaries
build and the
test engine<br>
starts:<br>
<br>
Platform
default build
- pass<br>
Simple RCF
consistency
check
succeeded<br>
--->>>
Starting
Logger...done<br>
--->>>
Starting
RCF...rcf_net_engine_connect():
Connection
timed<br>
out <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu:23571</a><br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"
moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a><br>
<br>
Then, it
hangs here
until I kill
the "te_rcf"
and "te_tee"<br>
processes.
I let it hang
for around 9
minutes.<br>
<br>
On the
tester host
(which appears
to be the Peer
agent), there
are<br>
four
processes that
I see running,
which look
like the test
agent<br>
processes.<br>
<br>
ta.Peer is
an empty file.
I've attached
the log.txt
from this run.<br>
<br>
- Adam<br>
<br>
On Thu,
Aug 24, 2023
at 4:22��AM
Andrew
Rybchenko<br>
<<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a><br>
<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote:<br>
<br>
Hi
Adam,<br>
<br>
Yes,
TE_RCFUNIX_TIMEOUT
is in seconds.
I've
double-checked<br>
that
it goes to
'copy_timeout'
in
ts-conf/rcf.conf.<br>
Description in
in
doc/sphinx/pages/group_te_engine_rcf.rst<br>
says
that
copy_timeout
is in seconds
and
implementation
in<br>
lib/rcfunix/rcfunix.c
passes the
value to
select()
tv_sec.<br>
Theoretically
select() could
be interrupted
by signal, but
I<br>
think
it is unlikely
here.<br>
<br>
I'm
not sure that
I understand
what do you
mean by RCF<br>
connection
timeout. Does
it happen on
TE startup
when RCF<br>
starts
test agents.
If so,
TE_RCFUNIX_TIMEOUT
could help. Or<br>
does
it happen when
tests are in
progress, e.g.
in the middle<br>
of a
test. If so,
TE_RCFUNIX_TIMEOUT
is unrelated
and most<br>
likely
either host
with test
agent dies or
test agent
itself<br>
crashes. It
would be
easier for me
if classify it
if you share<br>
text
log (log.txt,
full or just
corresponding
fragment with<br>
some
context). Also
content of
ta.DPDK or
ta.Peer file<br>
depending on
which agent
has problems
could shed
some light.<br>
Corresponding
files contain
stdout/stderr
of test
agents.<br>
<br>
Andrew.<br>
<br>
On
8/23/23 17:45,
Adam Hassick
wrote:<br>
<blockquote
type="cite">
Hi Andrew,<br>
<br>
I've
set up a test
rig repository
here, and have
created<br>
configurations
for our
development
testbed based
off of the<br>
examples.<br>
We've
been able to
get the test
suite to run
manually on<br>
Mellanox CX5
devices once.<br>
However, we
are running
into an issue
where, when
RCF starts,<br>
the
RCF connection
times out very
frequently. We
aren't sure<br>
why
this is the
case.<br>
It
works
sometimes, but
most of the
time when we
try to run<br>
the
test engine,
it encounters
this issue.<br>
I've
tried changing
the RCF port
by setting<br>
"TE_RCF_PORT=<some
port
number>"
and rebooting
the testbed<br>
machines.
Neither seems
to fix the
issue.<br>
<br>
It
also seems
like the
timeout takes
far longer
than 60<br>
seconds, even
when running
"export
TE_RCFUNIX_TIMEOUT=60"<br>
before
I try to run
the test
suite.<br>
I
assume the
unit for this
variable is
seconds?<br>
<br>
Thanks,<br>
Adam<br>
<br>
On
Mon, Aug 21,
2023 at
10:19 AM Adam
Hassick<br>
<<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>>
wrote:<br>
<br>
Hi
Andrew,<br>
<br>
Thanks, I've
cloned the
example
repository and
will start<br>
setting up a
configuration
for our
development
testbed<br>
today. I'll
let you know
if I run into
any
difficulties<br>
or
have any
questions.<br>
<br>
-
Adam<br>
<br>
On
Sun, Aug 20,
2023 at
4:40 AM Andrew
Rybchenko<br>
<<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a><br>
<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote:<br>
<br>
Hi Adam,<br>
<br>
I've published<br>
<a href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ts-factory/ts-rigs-sample</a><br>
<a href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/ts-rigs-sample></a>.<br>
Hopefully it will help to define your test rigs and<br>
successfully run some tests manually. Feel free to<br>
ask any questions and I'll answer here and try to<br>
update documentation.<br>
<br>
Meanwhile I'll prepare missing bits for steps (2) and<br>
(3).<br>
Hopefully everything is in place for step (4), but we<br>
need to make steps (2) and (3) first.<br>
<br>
Andrew.<br>
<br>
On 8/18/23 21:40, Andrew Rybchenko wrote:<br>
<blockquote
type="cite">
Hi Adam,<br>
<br>
> I've conferred with the rest of the team, and we<br>
think it would be best to move forward with mainly<br>
option B.<br>
<br>
OK, I'll provide the sample on Monday for you. It is<br>
almost ready right now, but I need to double-check<br>
it before publishing.<br>
<br>
Regards,<br>
Andrew.<br>
<br>
On 8/17/23 20:03, Adam Hassick wrote:<br>
<blockquote
type="cite">
Hi Andrew,<br>
<br>
I'm adding the CI mailing list to this<br>
conversation. Others in the community might find<br>
this conversation valuable.<br>
<br>
We do want to run testing on a regular basis. The<br>
Jenkins integration will be very useful for us, as<br>
most of our CI is orchestrated by Jenkins.<br>
I've conferred with the rest of the team, and we<br>
think it would be best to move forward with mainly<br>
option B.<br>
If you would like to know anything about our<br>
testbeds that would help you with creating an<br>
example ts-rigs repo, I'd be happy to answer any<br>
questions you have.<br>
<br>
We have multiple test rigs (we call these<br>
"DUT-tester pairs") that we run our existing<br>
hardware testing on, with differing network<br>
hardware and CPU architecture. I figured this might<br>
be an important detail.<br>
<br>
Thanks,<br>
Adam<br>
<br>
On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko<br>
<<a href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a><br>
<a href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote:<br>
<br>
Greatings Adam,<br>
<br>
I'm happy to hear that you're trying to bring<br>
it up.<br>
<br>
As I understand the final goal is to run it on<br>
regular basis. So, we need to make it properly<br>
from the very beginning.<br>
Bring up of all features consists of 4 steps:<br>
<br>
1. Create site-specific repository (we call it<br>
ts-rigs) which contains information about test<br>
rigs and other site-specific information like<br>
where to send mails, where to store logs etc.<br>
It is required for manual execution as well,<br>
since test rigs description is essential. I'll<br>
return to the topic below.<br>
<br>
2. Setup logs storage for automated runs.<br>
Basically it is a disk space plus apache2 web<br>
server with few CGI scripts which help a lot to<br>
save disk space.<br>
<br>
3. Setup Bublik web application which provides<br>
web interface to view testing results. Same as<br>
<a href="https://ts-factory.io/bublik"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://ts-factory.io/bublik</a><br>
<a href="https://ts-factory.io/bublik"
target="_blank" moz-do-not-send="true"><https://ts-factory.io/bublik></a><br>
<br>
4. Setup Jenkins to run tests on regularly,<br>
save logs in log storage (2) and import it to<br>
bublik (3).<br>
<br>
Last few month we spent on our homework to make<br>
it simpler to bring up automated execution<br>
using Jenkins -<br>
<a href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ts-factory/te-jenkins</a><br>
<a href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/te-jenkins></a><br>
Corresponding bits in dpdk-ethdev-ts will be<br>
available tomorrow.<br>
<br>
Let's return to the step (1).<br>
<br>
Unfortunately there is no publicly available<br>
example of the ts-rigs repository since<br>
sensitive site-specific information is located<br>
there. But I'm ready to help you to create it<br>
for UNH. I see two options here:<br>
<br>
(A) I'll ask questions and based on your<br>
answers will create the first draft with my<br>
comments.<br>
<br>
(B) I'll make a template/example ts-rigs repo,<br>
publish it and you'll create UNH ts-rigs based<br>
on it.<br>
<br>
Of course, I'll help to debug and finally bring<br>
it up in any case.<br>
<br>
(A) is a bit simpler for me and you, but (B) is<br>
a bit more generic and will help other<br>
potential users to bring it up.<br>
We can combine (A)+(B). I.e. start from (A).<br>
What do you think?<br>
<br>
Thanks,<br>
Andrew.<br>
<br>
On 8/17/23 15:18, Konstantin Ushakov wrote:<br>
<blockquote
type="cite">
Greetings
Adam,<br>
<br>
<br>
Thanks for contacting us. I copy Andrew who<br>
would be happy to help<br>
<br>
Thanks,<br>
Konstantin<br>
<br>
<blockquote
type="cite">
On 16 Aug
2023, at
21:50, Adam
Hassick<br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><ahassick@iol.unh.edu></a><br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
wrote:<br>
<br>
<br>
Greetings Konstantin,<br>
<br>
I am in the process of setting up the DPDK<br>
Poll Mode Driver test suite as an addition to<br>
our testing coverage for DPDK at the UNH lab.<br>
<br>
I have some questions about how to set the<br>
test suite arguments.<br>
<br>
I have been able to configure the Test Engine<br>
to connect to the hosts in the testbed. The<br>
RCF, Configurator, and Tester all begin to<br>
run, however the prelude of the test suite<br>
fails to run.<br>
<br>
<a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters</a>
<a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true"><https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters></a><br>
<br>
The documentation mentions that there are<br>
several test parameters for the test suite,<br>
like for the IUT test link MAC, etc. These<br>
seem like they would need to be set somewhere<br>
to run many of the tests.<br>
<br>
I see in the Test Engine documentation, there<br>
are instructions on how to create new<br>
parameters for test suites in the Tester<br>
configuration, but there is nothing in the<br>
user guide or in the Tester guide for how to<br>
set the arguments for the parameters when<br>
running the test suite that I can find. I'm<br>
not sure if I need to write my own Tester<br>
config, or if I should be setting these in<br>
some other way.<br>
<br>
How should these values be set?<br>
<br>
I'm also not sure what environment<br>
variables/arguments are strictly necessary or<br>
which are optional.<br>
<br>
Regards,<br>
Adam<br>
<br>
-- *Adam Hassick*<br>
Senior Developer<br>
UNH InterOperability Lab<br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a href="http://iol.unh.edu" target="_blank"
moz-do-not-send="true">iol.unh.edu</a> <a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1 (603) 475-8248<br>
</blockquote>
</blockquote>
<br>
<br>
<br>
-- *Adam Hassick*<br>
Senior Developer<br>
UNH InterOperability Lab<br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a href="http://iol.unh.edu" target="_blank"
moz-do-not-send="true">iol.unh.edu</a> <a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1 (603) 475-8248<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
<br>
--
*Adam Hassick*<br>
Senior
Developer<br>
UNH
InterOperability
Lab<br>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1
(603) 475-8248<br>
<br>
<br>
<br>
--
*Adam
Hassick*<br>
Senior
Developer<br>
UNH
InterOperability
Lab<br>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1
(603) 475-8248<br>
</blockquote>
<br>
<br>
<br>
--
*Adam Hassick*<br>
Senior
Developer<br>
UNH
InterOperability
Lab<br>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1 (603)
475-8248<br>
</blockquote>
<br>
<br>
<br>
-- <br>
*Adam Hassick*<br>
Senior
Developer<br>
UNH
InterOperability
Lab<br>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a><br>
<a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a><br>
+1 (603)
475-8248<br>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br
clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br
clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br
clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br
clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br
clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span
class="gmail_signature_prefix">--</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
</div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
<div
style="white-space:normal"> </div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span
class="gmail_signature_prefix">--
</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span
style="color:rgb(102,102,102)"></span></div>
<div><span
style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span
style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/"
target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span
style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span
style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/"
target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam
Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>