<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">> I'll double-check test engine on
Ubuntu 20.04 and Ubuntu 22.04.<br>
<br>
Done. It works fine for me without any issues.<br>
<br>
Have you noticed any pattern when it works or does not work?<br>
May be it is a problem of not clean state after termination?<br>
Does it work fine the first time after DUTs reboot?<br>
How do you terminate testing? It should be done using Ctrl+C in
terminal where you execute run.sh command.<br>
In this case it should shutdown gracefully and close all test
agents and engine applications.<br>
<br>
(I'm trying to understand why you've seen many test agent
processes. It should not happen.)<br>
<br>
Andrew.<br>
<br>
On 8/25/23 17:41, Andrew Rybchenko wrote:<br>
</div>
<blockquote type="cite"
cite="mid:6c9eea95-7d53-c82c-bced-823e4e9db62a@oktetlabs.ru">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">On 8/25/23 17:06, Adam Hassick wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAC-YWqj28X-YhLzM-rChuKMdaoTrDS6fN68YxzS+rb1N-i5AkQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div dir="ltr">
<div>
<div>Hi Andrew,<br>
<br>
</div>
Two of our systems (the Test Engine runner and the DUT host)
are running Ubuntu 20.04 LTS, however this morning I noticed
that the tester system (the one having issues) is running
Ubuntu 22.04 LTS.<br>
</div>
<div>This could be the source of the problem. I encountered a
dependency issue trying to run the Test Engine on 22.04 LTS,
so I downgraded the system. Since the tester is also the
host having connection issues, I will try downgrading that
system to 20.04, and see if that changes anything.<br>
</div>
</div>
</blockquote>
<br>
Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38
every night.<br>
Right now Debian 11 is used for test engine in nightly
regressions.<br>
<br>
I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.<br>
<br>
<blockquote type="cite"
cite="mid:CAC-YWqj28X-YhLzM-rChuKMdaoTrDS6fN68YxzS+rb1N-i5AkQ@mail.gmail.com">
<div dir="ltr">
<div>I did try passing in the "--vg-rcf" argument to the
run.sh script of the test suite after installing valgrind,
but there was no additional output that I saw.<br>
</div>
</div>
</blockquote>
<br>
Sorry, I should valgrind output should be in valgrind.te_rcf
(direction where you run test engine).<br>
<br>
<blockquote type="cite"
cite="mid:CAC-YWqj28X-YhLzM-rChuKMdaoTrDS6fN68YxzS+rb1N-i5AkQ@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>I will try pulling in the changes you've pushed up, and
will see if that fixes anything.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 25, 2023 at
9:57 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div>Hello Adam, <br>
<br>
On 8/24/23 23:54, Andrew Rybchenko wrote:<br>
</div>
<blockquote type="cite">I'd like to try to repeat the
problem locally. Which Linux distro is running on test
engine and agents? <br>
<br>
In fact I know one problem with Debian 12 and Fedora 38
and we have <br>
patch in review to fix it, however, the behaviour is
different in <br>
this case, so it is unlike the same problem. <br>
</blockquote>
<br>
I've just published a new tag which fixes known test
engine side problems on Debian 12 and Fedora 38.<br>
<br>
<blockquote type="cite"> <br>
One more idea is to install valgrind on the test engine
host and <br>
run with option --vg-rcf to check if something weird is
happening. <br>
<br>
What I don't understand right now is why I see just one
failed attempt <br>
to connect in your log.txt and then Logger shutdown
after 9 minutes. <br>
<br>
Andrew. <br>
<br>
On 8/24/23 23:29, Adam Hassick wrote: <br>
<blockquote type="cite"> > Is there any firewall in
the network or on test hosts which could block
incoming TCP connection to the port 23571 <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host where you run test engine? <br>
<br>
Our test engine host and the testbed are on the same
subnet. The connection does work sometimes. <br>
<br>
> If behaviour the same on the next try and you
see that test agent is kept running, could you check
using <br>
> <br>
> # netstat -tnlp <br>
> <br>
> that Test Agent is listening on the port and try
to establish TCP connection from test agent using <br>
> <br>
> $ telnet <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu"
target="_blank" moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571 <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
<br>
> <br>
> and check if TCP connection could be
established. <br>
<br>
I was able to replicate the same behavior again, where
it hangs while RCF is trying to start. <br>
Running this command, I see this in the output: <br>
<br>
tcp 0 0 <a href="http://0.0.0.0:23571"
target="_blank" moz-do-not-send="true">0.0.0.0:23571</a>
<a href="http://0.0.0.0:23571" target="_blank"
moz-do-not-send="true"><http://0.0.0.0:23571></a>
0.0.0.0:* LISTEN 18599/ta <br>
<br>
So it seems like it is listening on the correct port.
<br>
Additionally, I was able to connect to the Tester
machine from our Test Engine host using telnet. It
printed the PID of the process once the connection was
opened. <br>
<br>
I tried running the "ta" application manually on the
command line, and it didn't print anything at all. <br>
Maybe the issue is something on the Test Engine side.
<br>
<br>
On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
<a href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Hi Adam, <br>
<br>
> On the tester host (which appears to be the
Peer agent), there <br>
are four processes that I see running, which look
like the test <br>
agent processes. <br>
<br>
Before the next try I'd recommend to kill these
processes. <br>
<br>
Is there any firewall in the network or on test
hosts which could <br>
block incoming TCP connection to the port 23571 <br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host <br>
where you run test engine? <br>
<br>
If behaviour the same on the next try and you see
that test agent is <br>
kept running, could you check using <br>
<br>
# netstat -tnlp <br>
<br>
that Test Agent is listening on the port and try
to establish TCP <br>
connection from test agent using <br>
<br>
$ telnet <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu"
target="_blank" moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a>
<br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571 <br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
<br>
<br>
and check if TCP connection could be established.
<br>
<br>
Another idea is to login Tester under root as
testing does, get <br>
start TA command from the log and try it by hands
without -n and <br>
remove extra escaping. <br>
<br>
# sudo
PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1 <br>
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=<br>
<br>
Hopefully in this case test agent directory
remains in the /tmp and <br>
you don't need to copy it as testing does. <br>
May be output could shed some light on what's
going on. <br>
<br>
Andrew. <br>
<br>
On 8/24/23 17:30, Adam Hassick wrote: <br>
<blockquote type="cite"> Hi Andrew, <br>
<br>
This is the output that I see in the terminal
when this failure <br>
occurs, after the test agent binaries build and
the test engine <br>
starts: <br>
<br>
Platform default build - pass <br>
Simple RCF consistency check succeeded <br>
--->>> Starting Logger...done <br>
--->>> Starting
RCF...rcf_net_engine_connect(): Connection timed <br>
out <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu:23571</a>
<br>
<a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
<br>
<br>
Then, it hangs here until I kill the "te_rcf"
and "te_tee" <br>
processes. I let it hang for around 9 minutes. <br>
<br>
On the tester host (which appears to be the Peer
agent), there are <br>
four processes that I see running, which look
like the test agent <br>
processes. <br>
<br>
ta.Peer is an empty file. I've attached the
log.txt from this run. <br>
<br>
- Adam <br>
<br>
On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
<br>
<<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
<br>
<a href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Hi Adam, <br>
<br>
Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
double-checked <br>
that it goes to 'copy_timeout' in
ts-conf/rcf.conf. <br>
Description in in
doc/sphinx/pages/group_te_engine_rcf.rst <br>
says that copy_timeout is in seconds and
implementation in <br>
lib/rcfunix/rcfunix.c passes the value to
select() tv_sec. <br>
Theoretically select() could be interrupted
by signal, but I <br>
think it is unlikely here. <br>
<br>
I'm not sure that I understand what do you
mean by RCF <br>
connection timeout. Does it happen on TE
startup when RCF <br>
starts test agents. If so,
TE_RCFUNIX_TIMEOUT could help. Or <br>
does it happen when tests are in progress,
e.g. in the middle <br>
of a test. If so, TE_RCFUNIX_TIMEOUT is
unrelated and most <br>
likely either host with test agent dies or
test agent itself <br>
crashes. It would be easier for me if
classify it if you share <br>
text log (log.txt, full or just
corresponding fragment with <br>
some context). Also content of ta.DPDK or
ta.Peer file <br>
depending on which agent has problems could
shed some light. <br>
Corresponding files contain stdout/stderr of
test agents. <br>
<br>
Andrew. <br>
<br>
On 8/23/23 17:45, Adam Hassick wrote: <br>
<blockquote type="cite"> Hi Andrew, <br>
<br>
I've set up a test rig repository here,
and have created <br>
configurations for our development testbed
based off of the <br>
examples. <br>
We've been able to get the test suite to
run manually on <br>
Mellanox CX5 devices once. <br>
However, we are running into an issue
where, when RCF starts, <br>
the RCF connection times out very
frequently. We aren't sure <br>
why this is the case. <br>
It works sometimes, but most of the time
when we try to run <br>
the test engine, it encounters this issue.
<br>
I've tried changing the RCF port by
setting <br>
"TE_RCF_PORT=<some port number>" and
rebooting the testbed <br>
machines. Neither seems to fix the issue.
<br>
<br>
It also seems like the timeout takes far
longer than 60 <br>
seconds, even when running "export
TE_RCFUNIX_TIMEOUT=60" <br>
before I try to run the test suite. <br>
I assume the unit for this variable is
seconds? <br>
<br>
Thanks, <br>
Adam <br>
<br>
On Mon, Aug 21, 2023 at 10:19 AM Adam
Hassick <br>
<<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>>
wrote: <br>
<br>
Hi Andrew, <br>
<br>
Thanks, I've cloned the example
repository and will start <br>
setting up a configuration for our
development testbed <br>
today. I'll let you know if I run into
any difficulties <br>
or have any questions. <br>
<br>
- Adam <br>
<br>
On Sun, Aug 20, 2023 at 4:40 AM Andrew
Rybchenko <br>
<<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
<br>
<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Hi Adam, <br>
<br>
I've published <br>
<a
href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/ts-factory/ts-rigs-sample</a>
<br>
<a
href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/ts-rigs-sample></a>.
<br>
Hopefully it will help to define
your test rigs and <br>
successfully run some tests
manually. Feel free to <br>
ask any questions and I'll answer
here and try to <br>
update documentation. <br>
<br>
Meanwhile I'll prepare missing
bits for steps (2) and <br>
(3). <br>
Hopefully everything is in place
for step (4), but we <br>
need to make steps (2) and (3)
first. <br>
<br>
Andrew. <br>
<br>
On 8/18/23 21:40, Andrew Rybchenko
wrote: <br>
<blockquote type="cite"> Hi Adam, <br>
<br>
> I've conferred with the
rest of the team, and we <br>
think it would be best to move
forward with mainly <br>
option B. <br>
<br>
OK, I'll provide the sample on
Monday for you. It is <br>
almost ready right now, but I
need to double-check <br>
it before publishing. <br>
<br>
Regards, <br>
Andrew. <br>
<br>
On 8/17/23 20:03, Adam Hassick
wrote: <br>
<blockquote type="cite"> Hi
Andrew, <br>
<br>
I'm adding the CI mailing list
to this <br>
conversation. Others in the
community might find <br>
this conversation valuable. <br>
<br>
We do want to run testing on a
regular basis. The <br>
Jenkins integration will be
very useful for us, as <br>
most of our CI is orchestrated
by Jenkins. <br>
I've conferred with the rest
of the team, and we <br>
think it would be best to move
forward with mainly <br>
option B. <br>
If you would like to know
anything about our <br>
testbeds that would help you
with creating an <br>
example ts-rigs repo, I'd be
happy to answer any <br>
questions you have. <br>
<br>
We have multiple test rigs (we
call these <br>
"DUT-tester pairs") that we
run our existing <br>
hardware testing on, with
differing network <br>
hardware and CPU architecture.
I figured this might <br>
be an important detail. <br>
<br>
Thanks, <br>
Adam <br>
<br>
On Thu, Aug 17, 2023 at
11:44 AM Andrew Rybchenko <br>
<<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
<br>
<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Greatings Adam, <br>
<br>
I'm happy to hear that
you're trying to bring <br>
it up. <br>
<br>
As I understand the final
goal is to run it on <br>
regular basis. So, we need
to make it properly <br>
from the very beginning. <br>
Bring up of all features
consists of 4 steps: <br>
<br>
1. Create site-specific
repository (we call it <br>
ts-rigs) which contains
information about test <br>
rigs and other
site-specific information like <br>
where to send mails, where
to store logs etc. <br>
It is required for manual
execution as well, <br>
since test rigs
description is essential. I'll <br>
return to the topic below.
<br>
<br>
2. Setup logs storage for
automated runs. <br>
Basically it is a disk
space plus apache2 web <br>
server with few CGI
scripts which help a lot to <br>
save disk space. <br>
<br>
3. Setup Bublik web
application which provides <br>
web interface to view
testing results. Same as <br>
<a
href="https://ts-factory.io/bublik"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://ts-factory.io/bublik</a>
<br>
<a
href="https://ts-factory.io/bublik"
target="_blank" moz-do-not-send="true"><https://ts-factory.io/bublik></a>
<br>
<br>
4. Setup Jenkins to run
tests on regularly, <br>
save logs in log storage
(2) and import it to <br>
bublik (3). <br>
<br>
Last few month we spent on
our homework to make <br>
it simpler to bring up
automated execution <br>
using Jenkins - <br>
<a
href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/ts-factory/te-jenkins</a>
<br>
<a
href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/te-jenkins></a>
<br>
Corresponding bits in
dpdk-ethdev-ts will be <br>
available tomorrow. <br>
<br>
Let's return to the step
(1). <br>
<br>
Unfortunately there is no
publicly available <br>
example of the ts-rigs
repository since <br>
sensitive site-specific
information is located <br>
there. But I'm ready to
help you to create it <br>
for UNH. I see two options
here: <br>
<br>
(A) I'll ask questions and
based on your <br>
answers will create the
first draft with my <br>
comments. <br>
<br>
(B) I'll make a
template/example ts-rigs repo, <br>
publish it and you'll
create UNH ts-rigs based <br>
on it. <br>
<br>
Of course, I'll help to
debug and finally bring <br>
it up in any case. <br>
<br>
(A) is a bit simpler for
me and you, but (B) is <br>
a bit more generic and
will help other <br>
potential users to bring
it up. <br>
We can combine (A)+(B).
I.e. start from (A). <br>
What do you think? <br>
<br>
Thanks, <br>
Andrew. <br>
<br>
On 8/17/23 15:18,
Konstantin Ushakov wrote: <br>
<blockquote type="cite">
Greetings Adam, <br>
<br>
<br>
Thanks for contacting
us. I copy Andrew who <br>
would be happy to help <br>
<br>
Thanks, <br>
Konstantin <br>
<br>
<blockquote type="cite">
On 16 Aug 2023, at 21:50, Adam Hassick <br>
<a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><ahassick@iol.unh.edu></a>
<br>
<a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
wrote: <br>
<br>
<br>
Greetings Konstantin,
<br>
<br>
I am in the process of
setting up the DPDK <br>
Poll Mode Driver test
suite as an addition to <br>
our testing coverage
for DPDK at the UNH lab. <br>
<br>
I have some questions
about how to set the <br>
test suite arguments.
<br>
<br>
I have been able to
configure the Test Engine <br>
to connect to the
hosts in the testbed. The <br>
RCF, Configurator, and
Tester all begin to <br>
run, however the
prelude of the test suite <br>
fails to run. <br>
<br>
<a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters</a>
<a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true"><https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters></a>
<br>
<br>
The documentation
mentions that there are <br>
several test
parameters for the test suite, <br>
like for the IUT test
link MAC, etc. These <br>
seem like they would
need to be set somewhere <br>
to run many of the
tests. <br>
<br>
I see in the Test
Engine documentation, there <br>
are instructions on
how to create new <br>
parameters for test
suites in the Tester <br>
configuration, but
there is nothing in the <br>
user guide or in the
Tester guide for how to <br>
set the arguments for
the parameters when <br>
running the test suite
that I can find. I'm <br>
not sure if I need to
write my own Tester <br>
config, or if I should
be setting these in <br>
some other way. <br>
<br>
How should these
values be set? <br>
<br>
I'm also not sure what
environment <br>
variables/arguments
are strictly necessary or <br>
which are optional. <br>
<br>
Regards, <br>
Adam <br>
<br>
--
*Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability
Lab <br>
<a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<br>
<a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
<br>
<a
href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
</blockquote>
<br>
<br>
<br>
-- *Adam
Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
<br>
</blockquote>
<br>
<br>
<br>
-- *Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/" target="_blank"
moz-do-not-send="true"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
<br>
<br>
<br>
-- *Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/" target="_blank"
moz-do-not-send="true"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
<br>
<br>
<br>
-- *Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank"
moz-do-not-send="true">iol.unh.edu</a> <a
href="https://www.iol.unh.edu/" target="_blank"
moz-do-not-send="true"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
<br>
<br>
<br>
-- <br>
*Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank"
moz-do-not-send="true">iol.unh.edu</a> <a
href="https://www.iol.unh.edu/" target="_blank"
moz-do-not-send="true"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>