<div dir="ltr"><div><div><div><div><div>Hi Andrew,<br><br></div><div>Yes, I copied the X710 configs to set up XL710 configs. I changed the environment variable names from the X710 suffix to XL710 suffix in the script, and forgot to change them in the corresponding environment file.<br></div></div>That fixed the issue.<br><br></div>I got the checkup tests working on the XL710 now. Most of them are failing, which leads me to believe this is an issue with our testbed. Based on the DPDK documentation for i40e, the firmware and driver versions are much older than what DPDK 22.11 LTS and main prefer, so I'll try updating those.<br><br></div>For now I'm working on getting the XL710 checkup tests passing, and will pick up getting the E810 configured properly next. I'll let you know if I run into any more issues in relation to the test engine.<br><br></div><div>Thanks,<br></div><div>Adam<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 6, 2023 at 7:36 AM Andrew Rybchenko <<a href="mailto:andrew.rybchenko@oktetlabs.ru">andrew.rybchenko@oktetlabs.ru</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
On 9/5/23 18:01, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
The compilation warning issue is now resolved. Again,
thank you guys for fixing this for us. I can run the
tests on the Mellanox CX5s again, however I'm running
into a couple new issues with running the prologues on
the Intel cards.<br>
<br>
</div>
When running testing on the Intel XL710s, I see this
error appear in the log:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">ERROR prologue
Environment LIB 14:16:13.650<br>
Too few networks in available configuration (0) in
comparison with required (1)<br>
</blockquote>
<br>
</div>
This seems like a trivial configuration error, perhaps
this is something I need to set up in ts-rigs. I briefly
searched through the examples there and didn't see any
mention of how to set up a network.<br>
</div>
<div>I will attach this log just in case you need more
information.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Unfortunately logs are insufficient to understand it. I've pushed
new tag to TE v1.19.0 which add log message with TE_* environment
variables.<br>
Most likely something is wrong with variables which are used as
conditions when available networks are defined in
ts-conf/cs/inc.net_cfg_pci_fns.yml:<br>
TE_PCI_INSTANCE_IUT_TST1<br>
TE_PCI_INSTANCE_IUT_TST1a<br>
TE_PCI_INSTANCE_TST1a_IUT<br>
TE_PCI_INSTANCE_TST1_IUT<br>
My guess it that you change naming a bit, but script like
ts-rigs-sample/scripts/iut.h1-x710 is not included or not updated.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>There is a different error when running on the Intel
E810s. It appears to me like it starts DPDK, does some
configuration inside DPDK and on the device, and then fails
to bring the device back up. Since this error seems very
non-trivial, I will also attach this log.<br>
</div>
</div>
</div>
</blockquote>
<br>
This one is a bit simpler. Few lines after the first ERROR in log I
see the following:<br>
WARN RCF DPDK 13:06:00.144<br>
ice_program_hw_rx_queue(): currently package doesn't support RXDID
(22)<br>
ice_rx_queue_start(): fail to program RX queue 0<br>
ice_dev_start(): fail to start Rx queue 0<br>
Device with port_id=0 already stopped<br>
<br>
It is stdout/stderr from test agent which runs DPDK. Same logs in
plain format are available in ta.DPDK file.<br>
I'm not an expert here, but I vaguely remember that E810 requires
correct firmware and DDP to be loaded.<br>
There is some information in dpdk/doc/guides/nics/ice.rst.<br>
<br>
You can try to add --dev-args=safe-mode-support=1 command-line
option described there.<br>
<br>
Hope it helps,<br>
Andrew.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div><br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Sep 1, 2023 at 3:59 AM
Andrew Rybchenko <<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
On 8/31/23 22:38, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Andrew,<br>
</div>
<div><br>
I have one additional question as well: Does the test
engine support running tests on two ARMv8 test agents?</div>
<div><br>
</div>
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">1. We'll sort out
warnings this week. Thanks for heads up.<br>
</blockquote>
<div><br>
</div>
<div>Great. Let me know when that's fixed.</div>
</div>
</div>
</blockquote>
<br>
Done. We also fixed a number of warnings in TE.<br>
Also we fixed root test package name to be consistent with
the repository name.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>Support for old LTS branches was dropped some
time ago, but in the future it is definitely
possible to keep it for new LTS branches. I think
22.11 is supported, but I'm not sure about older LTS
releases.</div>
</blockquote>
<div><br>
</div>
<div>Good to know.<br>
<div> <br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> 2. You can add
command-line option --sanity to run tests marked
with TEST_HARNESS_SANITY requirement (see
dpdk-ethdev-ts/scripts/run.sh and grep
TEST_HARNESS_SANITY dpdk-ethdev-ts to see which
tests are marked). Yes, there is a space for
terminology improvement here. We'll do it.<br>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
Done. Now it is called --checkup.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Also it takes a lot of time because of failures
and tests which wait for some timeout.<br>
</blockquote>
</div>
<div><br>
</div>
<div>That makes sense to me. We'll use the time to
complete tests on virtio or the Intel devices as a
reference for how long the tests really take to
complete.<br>
</div>
<div>We will explore the possibility of periodically
running the sanity tests for patches.<br>
</div>
</div>
</div>
</blockquote>
<br>
I'll double-check and let you know how long entire TS runs
on Intel X710, E810, Mellanox CX5 and virtio net. Just to
ensure that time observed in your case looks the same.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div> <br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> The test
harness can provide coverage reports based on
gcov, but I'm not sure what you mean by a "dial"
to control test coverage. Provided reports are
rather for human to analyze.<br>
</blockquote>
</div>
<div><br>
</div>
<div>The general idea is to have some kind of
parameter on the test suite, which could be an
integer ranging from zero to ten, that controls how
many tests are run based on how important the test
is.<br>
<br>
</div>
<div>Similar to how some command line interfaces
provide a verbosity level parameter (some number of
"-v" arguments) to control the importance of the
information in the log.<br>
</div>
The verbosity level zero only prints very important
log messages, while ten prints everything.<br>
</div>
<div><br>
In much the same manner as above, this "dial"
parameter controls what tests are run and with what
parameters based on how important those tests and test
parameter combinations are.<br>
Coverage Level zero tells the suite to run a very
basic set of important tests, with minimal
parameterization. This mode would take only ~5-10
minutes to run.<br>
In contrast, Coverage Level ten includes all the edge
cases, every combination of test parameters,
everything the test suite can do, which takes the
normal several hours to run.<br>
The values 1 - 9 are between those two extremes,
allowing the user to get a gradient of test coverage
in the results and to limit the running time.<br>
<br>
</div>
Then we could, for example, run the "run.sh" with a
level of 2 or 3 for incoming patches that need quick
results, and with a level of 10 for the less often run
periodic tests performed on main or LTS branches.<br>
</div>
</blockquote>
<br>
Understood now. Thanks a lot for the idea. We'll discuss it
and come back.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div> 3. Yes, really many tests on Mellanox CX5
NICs report unexpected testing results.
Unfortunately it is time consuming to fill in
expectations database since it is necessary to
analyze testing results and classify if it is
a bug or just acceptable behaviour aspect.<br>
<br>
Bublik allows to compare results of two runs.
It is useful for human, but still not good for
automation.<br>
<br>
I have local patch for mlx5 driver which
reports Tx ring size maximum. It makes pass
rate higher. It is a problem for test harness
that mlx5 does not report limits right now.<br>
<br>
Pass rate on Intel X710 is about 92% on my
test rig. Pass rate on virtio net is 99% right
now and could be done 100% easily (just one
thing to fix in expectations).<br>
<br>
I think logs storage setup is essential for
logs analysis. Of course, you can request HTML
logs when you run tests (--log-html=html) or
generate after run using
dpdk-ethdev-ts/scripts/html-log.sh and open
index.html in a browser, but logs storage
makes it more convenient.<br>
</div>
</div>
</blockquote>
<div><br>
We are interested in setting up Bublik,
potentially as an externally-facing component,
once we have our process of running the test suite
stabilized.</div>
<div>Once we are able to run the test suite again,
I'll see what the pass rate is on our other
hardware.<br>
Good to know that it isn't an issue with our dev
testbed causing the high fail rate.</div>
</div>
<div>
<div><br>
</div>
<div>For Intel hardware, we have an XL710 and an
Intel E810-C in our development testbed. Although
they are slightly different devices, ideally the
pass rate will be identical or similar. I have yet
to set up a VM pair for virtio, but we will soon.<br>
</div>
<div><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Latest
version of test-environment has examples of our
CGI scripts which we use for log storage (see
tools/log_server/README.md).<br>
<br>
Also all bits for Jenkins setup are available.
See dpdk-ethdev-ts/jenkins/README.md and
examples of jenkins files in ts-rigs-sample.<br>
</blockquote>
</div>
<div><br>
</div>
<div>Jenkins integration, setting up production rig
configurations, and permanent log storage will be
our next steps once I am able to run the tests
again.<br>
</div>
<div>Unless there is an easy way to have meson not
pass "-Werror" into GCC. Then I would be able to
run the test suite.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Hopefully it is resolved now.<br>
<br>
I thought a bit more about your usecase for Jenkins. I'm not
100% sure that existing pipelines are convenient for your
usecase.<br>
Fill free to ask questions when you are on it.<br>
<br>
Thanks,<br>
Andrew.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div> <br>
On 8/29/23 17:02, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
That fix seems to have resolved the
issue, thanks for the quick turnaround
time on that patch.<br>
</div>
<div>Now that we have the RCF timeout
issue resolved, there are a few other
questions and issues that we have about
the tests themselves.</div>
<br>
</div>
<div>1. The test suite fails to build with a
couple warnings.<br>
</div>
<div><br>
</div>
<div>Below is the stderr log from
compilation:<br>
</div>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">FAILED:
<a href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o" target="_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
-I../../lib
-I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
-fdiagnostics-color=always -pipe
-D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
-Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall
-W -fPIC -MD -MQ '<a href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o" target="_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-MF '<a href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d" target="_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
-o '<a href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o" target="_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-c ../../lib/dpdk_pmd_ts.c<br>
../../lib/dpdk_pmd_ts.c: In function
‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error:
format not a string literal and no format
arguments [-Werror=format-security]<br>
5577 | rc = te_kvpair_add(result, buf,
mode);<br>
| ^~<br>
cc1: all warnings being treated as errors<br>
ninja: build stopped: subcommand failed.<br>
ninja: Entering directory `.'<br>
FAILED: <a href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o" target="_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib
-I../../lib
-I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
-fdiagnostics-color=always -pipe
-D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
-Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall
-W -fPIC -MD -MQ '<a href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o" target="_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-MF '<a href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d" target="_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
-o '<a href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o" target="_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
-c ../../lib/dpdk_pmd_ts.c<br>
../../lib/dpdk_pmd_ts.c: In function
‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error:
format not a string literal and no format
arguments [-Werror=format-security]<br>
5577 | rc = te_kvpair_add(result, buf,
mode);<br>
| ^~<br>
cc1: all warnings being treated as errors<br>
</blockquote>
<div>
<div>
<div><br>
</div>
<div>This error wasn't occurring last
week, which was the last time I ran
the tests.<br>
</div>
<div>The TE host and the DUT have GCC
v9.4.0 installed, and the tester has
GCC v11.4.0 installed, if this
information is helpful.<br>
</div>
<div><br>
</div>
<div>2. On the Mellanox CX5s, there are
over 6,000 tests run, which
collectively take around 9 hours. Is
it possible, and would it make sense,
to lower the test coverage and have
the test suite run faster?<br>
<br>
</div>
<div>For some context, we run immediate
testing on incoming patches for DPDK
main and development branches, as well
as periodic test runs on the main,
stable, and LTS branches.<br>
</div>
<div>For us to consider including this
test suite as part of our immediate
testing on patches, we would have to
reduce the test coverage to the most
important tests.<br>
This is primarily to reduce the
testing time to, for example, less
than 30 minutes. Testing on patches
can't take too long because the lab
can receive numerous patches each day,
which each require individual testing
runs.<br>
<br>
</div>
<div>At what frequency we run these
tests, and on what, still needs to be
discussed with the DPDK community, but
it would be nice to know if the test
suite had a "dial" to control the
testing coverage.<br>
</div>
<div><br>
</div>
<div>3. We see a lot of test failures on
our Mellanox CX5 NICs. Around 2,300 of
~6,600 tests passed. Is there anything
we can do to diagnose these test
failures?<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue,
Aug 29, 2023 at 8:07 AM Andrew Rybchenko
<<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
I've pushed the fix in main branch and
a new tag v1.18.1. It should solve the
problem with IPv6 address from DNS.<br>
<br>
Andrew.<br>
<br>
On 8/29/23 00:05, Andrew Rybchenko
wrote:<br>
</div>
<blockquote type="cite">
<div>Hi Adam,<br>
<br>
> Does the test engine prefer to
use IPv6 over IPv4 for initiating
the RCF connection to the test bed
hosts? And if so, is there a way to
force it to use IPv4?<br>
<br>
Brilliant idea. If DNS returns both
IPv4 and IPv6 addresses in your
case, I guess it is the root cause
of the problem.<br>
Of course, it is TE problem since I
see really weird code in
lib/comm_net_engine/comm_net_engine.c
line 135.<br>
<br>
I've pushed fix to the branch
user/arybchik/fix_ipv4_only in
ts-factory/test-environment
repository. Please, try.<br>
<br>
It is late night fix with minimal
testing and no review. I'll pass it
through review process tomorrow and<br>
hopefully it will be released in
one-two days.<br>
<br>
Andrew.<br>
<br>
On 8/28/23 18:02, Adam Hassick
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
We have yet to notice a
distinct pattern with the
failures. Sometimes, the RCF
will start and connect without
issue a few times in a row
before failing to connect
again. Once the issue begins
to occur, neither rebooting
all of the hosts (test engine
VM, tester, IUT) or deleting
all of the build directories
(suites, agents, inst) and
rebooting the hosts afterward
resolves the issue. When it
begins working again seems
very arbitrary to us.<br>
<br>
</div>
<div>I do usually try to
terminate the test engine with
Ctrl+C, but when it hangs
while trying to start RCF,
that does not work.<br>
</div>
<div><br>
</div>
<div>Does the test engine prefer
to use IPv6 over IPv4 for
initiating the RCF connection
to the test bed hosts? And if
so, is there a way to force it
to use IPv4?<br>
<br>
</div>
<div> - Adam<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Fri, Aug 25, 2023 at 1:35 PM
Andrew Rybchenko <<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>> I'll double-check
test engine on Ubuntu 20.04
and Ubuntu 22.04.<br>
<br>
Done. It works fine for me
without any issues.<br>
<br>
Have you noticed any pattern
when it works or does not
work?<br>
May be it is a problem of
not clean state after
termination?<br>
Does it work fine the first
time after DUTs reboot?<br>
How do you terminate
testing? It should be done
using Ctrl+C in terminal
where you execute run.sh
command.<br>
In this case it should
shutdown gracefully and
close all test agents and
engine applications.<br>
<br>
(I'm trying to understand
why you've seen many test
agent processes. It should
not happen.)<br>
<br>
Andrew.<br>
<br>
On 8/25/23 17:41, Andrew
Rybchenko wrote:<br>
</div>
<blockquote type="cite">
<div>On 8/25/23 17:06, Adam
Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Andrew,<br>
<br>
</div>
Two of our systems
(the Test Engine
runner and the DUT
host) are running
Ubuntu 20.04 LTS,
however this morning I
noticed that the
tester system (the one
having issues) is
running Ubuntu 22.04
LTS.<br>
</div>
<div>This could be the
source of the problem.
I encountered a
dependency issue
trying to run the Test
Engine on 22.04 LTS,
so I downgraded the
system. Since the
tester is also the
host having connection
issues, I will try
downgrading that
system to 20.04, and
see if that changes
anything.<br>
</div>
</div>
</blockquote>
<br>
Unlikely, but who knows. We
run tests (DUTs) on Ubuntu
20.04, Ubuntu 22.04, Ubuntu
22.10, Ubuntu 23.04, Debian
11 and Fedora 38 every
night.<br>
Right now Debian 11 is used
for test engine in nightly
regressions.<br>
<br>
I'll double-check test
engine on Ubuntu 20.04 and
Ubuntu 22.04.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>I did try passing
in the "--vg-rcf"
argument to the run.sh
script of the test
suite after installing
valgrind, but there
was no additional
output that I saw.<br>
</div>
</div>
</blockquote>
<br>
Sorry, I should valgrind
output should be in
valgrind.te_rcf (direction
where you run test engine).<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I will try pulling
in the changes you've
pushed up, and will
see if that fixes
anything.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Fri, Aug 25, 2023 at
9:57 AM Andrew
Rybchenko <<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hello Adam, <br>
<br>
On 8/24/23 23:54,
Andrew Rybchenko
wrote:<br>
</div>
<blockquote type="cite">I'd
like to try to
repeat the problem
locally. Which
Linux distro is
running on test
engine and agents?
<br>
<br>
In fact I know one
problem with
Debian 12 and
Fedora 38 and we
have <br>
patch in review to
fix it, however,
the behaviour is
different in <br>
this case, so it
is unlike the same
problem. <br>
</blockquote>
<br>
I've just published
a new tag which
fixes known test
engine side problems
on Debian 12 and
Fedora 38.<br>
<br>
<blockquote type="cite"> <br>
One more idea is
to install
valgrind on the
test engine host
and <br>
run with option
--vg-rcf to check
if something weird
is happening. <br>
<br>
What I don't
understand right
now is why I see
just one failed
attempt <br>
to connect in your
log.txt and then
Logger shutdown
after 9 minutes. <br>
<br>
Andrew. <br>
<br>
On 8/24/23 23:29,
Adam Hassick
wrote: <br>
<blockquote type="cite"> >
Is there any
firewall in the
network or on
test hosts which
could block
incoming TCP
connection to
the port 23571 <a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host
where you run
test engine? <br>
<br>
Our test engine
host and the
testbed are on
the same subnet.
The connection
does work
sometimes. <br>
<br>
> If
behaviour the
same on the next
try and you see
that test agent
is kept running,
could you check
using <br>
> <br>
> # netstat
-tnlp <br>
> <br>
> that Test
Agent is
listening on the
port and try to
establish TCP
connection from
test agent using
<br>
> <br>
> $ telnet <a href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank">iol-dts-tester.dpdklab.iol.unh.edu</a>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571 <a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
<br>
> <br>
> and check
if TCP
connection could
be established.
<br>
<br>
I was able to
replicate the
same behavior
again, where it
hangs while RCF
is trying to
start. <br>
Running this
command, I see
this in the
output: <br>
<br>
tcp 0
0 <a href="http://0.0.0.0:23571" target="_blank">0.0.0.0:23571</a> <a href="http://0.0.0.0:23571" target="_blank"><http://0.0.0.0:23571></a>
0.0.0.0:*
LISTEN
18599/ta <br>
<br>
So it seems like
it is listening
on the correct
port. <br>
Additionally, I
was able to
connect to the
Tester machine
from our Test
Engine host
using telnet. It
printed the PID
of the process
once the
connection was
opened. <br>
<br>
I tried running
the "ta"
application
manually on the
command line,
and it didn't
print anything
at all. <br>
Maybe the issue
is something on
the Test Engine
side. <br>
<br>
On Thu, Aug 24,
2023 at 2:35 PM
Andrew Rybchenko
<<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>
<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Hi Adam, <br>
<br>
> On the
tester host
(which appears
to be the Peer
agent), there <br>
are four
processes that I
see running,
which look like
the test <br>
agent
processes. <br>
<br>
Before the
next try I'd
recommend to
kill these
processes. <br>
<br>
Is there any
firewall in the
network or on
test hosts which
could <br>
block
incoming TCP
connection to
the port 23571 <br>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host <br>
where you
run test engine?
<br>
<br>
If behaviour
the same on the
next try and you
see that test
agent is <br>
kept
running, could
you check using
<br>
<br>
# netstat
-tnlp <br>
<br>
that Test
Agent is
listening on the
port and try to
establish TCP <br>
connection
from test agent
using <br>
<br>
$ telnet <a href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank">iol-dts-tester.dpdklab.iol.unh.edu</a>
<br>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571 <br>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
<br>
<br>
and check if
TCP connection
could be
established. <br>
<br>
Another idea
is to login
Tester under
root as testing
does, get <br>
start TA
command from the
log and try it
by hands without
-n and <br>
remove extra
escaping. <br>
<br>
# sudo
PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
<br>
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=<br>
<br>
Hopefully in
this case test
agent directory
remains in the
/tmp and <br>
you don't
need to copy it
as testing does.
<br>
May be
output could
shed some light
on what's going
on. <br>
<br>
Andrew. <br>
<br>
On 8/24/23
17:30, Adam
Hassick wrote: <br>
<blockquote type="cite">
Hi Andrew, <br>
<br>
This is
the output
that I see in
the terminal
when this
failure <br>
occurs,
after the test
agent binaries
build and the
test engine <br>
starts: <br>
<br>
Platform
default build
- pass <br>
Simple RCF
consistency
check
succeeded <br>
--->>>
Starting
Logger...done
<br>
--->>>
Starting
RCF...rcf_net_engine_connect():
Connection
timed <br>
out <a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank">iol-dts-tester.dpdklab.iol.unh.edu:23571</a>
<br>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
<br>
<br>
Then, it
hangs here
until I kill
the "te_rcf"
and "te_tee" <br>
processes.
I let it hang
for around 9
minutes. <br>
<br>
On the
tester host
(which appears
to be the Peer
agent), there
are <br>
four
processes that
I see running,
which look
like the test
agent <br>
processes.
<br>
<br>
ta.Peer is
an empty file.
I've attached
the log.txt
from this run.
<br>
<br>
- Adam <br>
<br>
On Thu,
Aug 24, 2023
at 4:22 AM
Andrew
Rybchenko <br>
<<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>
<br>
<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Hi
Adam, <br>
<br>
Yes,
TE_RCFUNIX_TIMEOUT
is in seconds.
I've
double-checked
<br>
that
it goes to
'copy_timeout'
in
ts-conf/rcf.conf.
<br>
Description in
in
doc/sphinx/pages/group_te_engine_rcf.rst
<br>
says
that
copy_timeout
is in seconds
and
implementation
in <br>
lib/rcfunix/rcfunix.c
passes the
value to
select()
tv_sec. <br>
Theoretically
select() could
be interrupted
by signal, but
I <br>
think
it is unlikely
here. <br>
<br>
I'm
not sure that
I understand
what do you
mean by RCF <br>
connection
timeout. Does
it happen on
TE startup
when RCF <br>
starts
test agents.
If so,
TE_RCFUNIX_TIMEOUT
could help. Or
<br>
does
it happen when
tests are in
progress, e.g.
in the middle
<br>
of a
test. If so,
TE_RCFUNIX_TIMEOUT
is unrelated
and most <br>
likely
either host
with test
agent dies or
test agent
itself <br>
crashes. It
would be
easier for me
if classify it
if you share <br>
text
log (log.txt,
full or just
corresponding
fragment with
<br>
some
context). Also
content of
ta.DPDK or
ta.Peer file <br>
depending on
which agent
has problems
could shed
some light. <br>
Corresponding
files contain
stdout/stderr
of test
agents. <br>
<br>
Andrew. <br>
<br>
On
8/23/23 17:45,
Adam Hassick
wrote: <br>
<blockquote type="cite">
Hi Andrew, <br>
<br>
I've
set up a test
rig repository
here, and have
created <br>
configurations
for our
development
testbed based
off of the <br>
examples. <br>
We've
been able to
get the test
suite to run
manually on <br>
Mellanox CX5
devices once.
<br>
However, we
are running
into an issue
where, when
RCF starts, <br>
the
RCF connection
times out very
frequently. We
aren't sure <br>
why
this is the
case. <br>
It
works
sometimes, but
most of the
time when we
try to run <br>
the
test engine,
it encounters
this issue. <br>
I've
tried changing
the RCF port
by setting <br>
"TE_RCF_PORT=<some
port
number>"
and rebooting
the testbed <br>
machines.
Neither seems
to fix the
issue. <br>
<br>
It
also seems
like the
timeout takes
far longer
than 60 <br>
seconds, even
when running
"export
TE_RCFUNIX_TIMEOUT=60"
<br>
before
I try to run
the test
suite. <br>
I
assume the
unit for this
variable is
seconds? <br>
<br>
Thanks, <br>
Adam <br>
<br>
On
Mon, Aug 21,
2023 at
10:19 AM Adam
Hassick <br>
<<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a> <a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>>
wrote: <br>
<br>
Hi
Andrew, <br>
<br>
Thanks, I've
cloned the
example
repository and
will start <br>
setting up a
configuration
for our
development
testbed <br>
today. I'll
let you know
if I run into
any
difficulties <br>
or
have any
questions. <br>
<br>
-
Adam <br>
<br>
On
Sun, Aug 20,
2023 at
4:40 AM Andrew
Rybchenko <br>
<<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>
<br>
<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Hi Adam, <br>
<br>
I've published <br>
<a href="https://github.com/ts-factory/ts-rigs-sample" target="_blank">https://github.com/ts-factory/ts-rigs-sample</a>
<br>
<a href="https://github.com/ts-factory/ts-rigs-sample" target="_blank"><https://github.com/ts-factory/ts-rigs-sample></a>.
<br>
Hopefully it will help to define your test rigs and <br>
successfully run some tests manually. Feel free to <br>
ask any questions and I'll answer here and try to <br>
update documentation. <br>
<br>
Meanwhile I'll prepare missing bits for steps (2) and <br>
(3). <br>
Hopefully everything is in place for step (4), but we <br>
need to make steps (2) and (3) first. <br>
<br>
Andrew. <br>
<br>
On 8/18/23 21:40, Andrew Rybchenko wrote: <br>
<blockquote type="cite">
Hi Adam, <br>
<br>
> I've conferred with the rest of the team, and we <br>
think it would be best to move forward with mainly <br>
option B. <br>
<br>
OK, I'll provide the sample on Monday for you. It is <br>
almost ready right now, but I need to double-check <br>
it before publishing. <br>
<br>
Regards, <br>
Andrew. <br>
<br>
On 8/17/23 20:03, Adam Hassick wrote: <br>
<blockquote type="cite">
Hi Andrew, <br>
<br>
I'm adding the CI mailing list to this <br>
conversation. Others in the community might find <br>
this conversation valuable. <br>
<br>
We do want to run testing on a regular basis. The <br>
Jenkins integration will be very useful for us, as <br>
most of our CI is orchestrated by Jenkins. <br>
I've conferred with the rest of the team, and we <br>
think it would be best to move forward with mainly <br>
option B. <br>
If you would like to know anything about our <br>
testbeds that would help you with creating an <br>
example ts-rigs repo, I'd be happy to answer any <br>
questions you have. <br>
<br>
We have multiple test rigs (we call these <br>
"DUT-tester pairs") that we run our existing <br>
hardware testing on, with differing network <br>
hardware and CPU architecture. I figured this might <br>
be an important detail. <br>
<br>
Thanks, <br>
Adam <br>
<br>
On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <br>
<<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>
<br>
<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Greatings Adam, <br>
<br>
I'm happy to hear that you're trying to bring <br>
it up. <br>
<br>
As I understand the final goal is to run it on <br>
regular basis. So, we need to make it properly <br>
from the very beginning. <br>
Bring up of all features consists of 4 steps: <br>
<br>
1. Create site-specific repository (we call it <br>
ts-rigs) which contains information about test <br>
rigs and other site-specific information like <br>
where to send mails, where to store logs etc. <br>
It is required for manual execution as well, <br>
since test rigs description is essential. I'll <br>
return to the topic below. <br>
<br>
2. Setup logs storage for automated runs. <br>
Basically it is a disk space plus apache2 web <br>
server with few CGI scripts which help a lot to <br>
save disk space. <br>
<br>
3. Setup Bublik web application which provides <br>
web interface to view testing results. Same as <br>
<a href="https://ts-factory.io/bublik" target="_blank">https://ts-factory.io/bublik</a>
<br>
<a href="https://ts-factory.io/bublik" target="_blank"><https://ts-factory.io/bublik></a> <br>
<br>
4. Setup Jenkins to run tests on regularly, <br>
save logs in log storage (2) and import it to <br>
bublik (3). <br>
<br>
Last few month we spent on our homework to make <br>
it simpler to bring up automated execution <br>
using Jenkins - <br>
<a href="https://github.com/ts-factory/te-jenkins" target="_blank">https://github.com/ts-factory/te-jenkins</a>
<br>
<a href="https://github.com/ts-factory/te-jenkins" target="_blank"><https://github.com/ts-factory/te-jenkins></a>
<br>
Corresponding bits in dpdk-ethdev-ts will be <br>
available tomorrow. <br>
<br>
Let's return to the step (1). <br>
<br>
Unfortunately there is no publicly available <br>
example of the ts-rigs repository since <br>
sensitive site-specific information is located <br>
there. But I'm ready to help you to create it <br>
for UNH. I see two options here: <br>
<br>
(A) I'll ask questions and based on your <br>
answers will create the first draft with my <br>
comments. <br>
<br>
(B) I'll make a template/example ts-rigs repo, <br>
publish it and you'll create UNH ts-rigs based <br>
on it. <br>
<br>
Of course, I'll help to debug and finally bring <br>
it up in any case. <br>
<br>
(A) is a bit simpler for me and you, but (B) is <br>
a bit more generic and will help other <br>
potential users to bring it up. <br>
We can combine (A)+(B). I.e. start from (A). <br>
What do you think? <br>
<br>
Thanks, <br>
Andrew. <br>
<br>
On 8/17/23 15:18, Konstantin Ushakov wrote: <br>
<blockquote type="cite">
Greetings
Adam, <br>
<br>
<br>
Thanks for contacting us. I copy Andrew who <br>
would be happy to help <br>
<br>
Thanks, <br>
Konstantin <br>
<br>
<blockquote type="cite">
On 16 Aug
2023, at
21:50, Adam
Hassick <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><ahassick@iol.unh.edu></a> <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a> wrote: <br>
<br>
<br>
Greetings Konstantin, <br>
<br>
I am in the process of setting up the DPDK <br>
Poll Mode Driver test suite as an addition to <br>
our testing coverage for DPDK at the UNH lab. <br>
<br>
I have some questions about how to set the <br>
test suite arguments. <br>
<br>
I have been able to configure the Test Engine <br>
to connect to the hosts in the testbed. The <br>
RCF, Configurator, and Tester all begin to <br>
run, however the prelude of the test suite <br>
fails to run. <br>
<br>
<a href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters" target="_blank">https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters</a>
<a href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters" target="_blank"><https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters></a>
<br>
<br>
The documentation mentions that there are <br>
several test parameters for the test suite, <br>
like for the IUT test link MAC, etc. These <br>
seem like they would need to be set somewhere <br>
to run many of the tests. <br>
<br>
I see in the Test Engine documentation, there <br>
are instructions on how to create new <br>
parameters for test suites in the Tester <br>
configuration, but there is nothing in the <br>
user guide or in the Tester guide for how to <br>
set the arguments for the parameters when <br>
running the test suite that I can find. I'm <br>
not sure if I need to write my own Tester <br>
config, or if I should be setting these in <br>
some other way. <br>
<br>
How should these values be set? <br>
<br>
I'm also not sure what environment <br>
variables/arguments are strictly necessary or <br>
which are optional. <br>
<br>
Regards, <br>
Adam <br>
<br>
-- *Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a> <br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
</blockquote>
<br>
<br>
<br>
-- *Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a> <a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
<br>
</blockquote>
<br>
<br>
<br>
--
*Adam Hassick*
<br>
Senior
Developer <br>
UNH
InterOperability
Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a> <a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1
(603) 475-8248
<br>
<br>
<br>
<br>
--
*Adam
Hassick* <br>
Senior
Developer <br>
UNH
InterOperability
Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a> <a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1
(603) 475-8248
<br>
</blockquote>
<br>
<br>
<br>
--
*Adam Hassick*
<br>
Senior
Developer <br>
UNH
InterOperability
Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a> <a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603)
475-8248 <br>
</blockquote>
<br>
<br>
<br>
-- <br>
*Adam Hassick* <br>
Senior Developer
<br>
UNH
InterOperability
Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a> <a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603)
475-8248 <br>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">--
</span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span style="background-color:rgb(255,255,255)"><span style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/" target="_blank">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">--
</span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span style="background-color:rgb(255,255,255)"><span style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/" target="_blank">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span style="background-color:rgb(255,255,255)"><span style="color:rgb(102,102,102)">Adam
Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/" target="_blank">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span style="background-color:rgb(255,255,255)"><span style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/" target="_blank">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><b><span style="background-color:rgb(255,255,255)"><span style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br></div><span style="color:rgb(102,102,102)"></span></div><div><span style="color:rgb(102,102,102)">Senior Developer</span></div><div><span style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span><span style="color:rgb(102,102,102)"></span></div><div><span style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a><br></span></div><div><span style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/" target="_blank">iol.unh.edu</a><br></span></div>+1 (603) 475-8248<br></div></div>