<div dir="ltr"><div><div><div>Hi Andrew,<br><br></div>We have yet to notice a distinct pattern with the failures. Sometimes, the RCF will start and connect without issue a few times in a row before failing to connect again. Once the issue begins to occur, neither rebooting all of the hosts (test engine VM, tester, IUT) or deleting all of the build directories (suites, agents, inst) and rebooting the hosts afterward resolves the issue. When it begins working again seems very arbitrary to us.<br><br></div><div>I do usually try to terminate the test engine with Ctrl+C, but when it hangs while trying to start RCF, that does not work.<br></div><div><br></div><div>Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF connection to the test bed hosts? And if so, is there a way to force it to use IPv4?<br><br></div><div> - Adam<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 25, 2023 at 1:35 PM Andrew Rybchenko <<a href="mailto:andrew.rybchenko@oktetlabs.ru">andrew.rybchenko@oktetlabs.ru</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>> I'll double-check test engine on
Ubuntu 20.04 and Ubuntu 22.04.<br>
<br>
Done. It works fine for me without any issues.<br>
<br>
Have you noticed any pattern when it works or does not work?<br>
May be it is a problem of not clean state after termination?<br>
Does it work fine the first time after DUTs reboot?<br>
How do you terminate testing? It should be done using Ctrl+C in
terminal where you execute run.sh command.<br>
In this case it should shutdown gracefully and close all test
agents and engine applications.<br>
<br>
(I'm trying to understand why you've seen many test agent
processes. It should not happen.)<br>
<br>
Andrew.<br>
<br>
On 8/25/23 17:41, Andrew Rybchenko wrote:<br>
</div>
<blockquote type="cite">
<div>On 8/25/23 17:06, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Andrew,<br>
<br>
</div>
Two of our systems (the Test Engine runner and the DUT host)
are running Ubuntu 20.04 LTS, however this morning I noticed
that the tester system (the one having issues) is running
Ubuntu 22.04 LTS.<br>
</div>
<div>This could be the source of the problem. I encountered a
dependency issue trying to run the Test Engine on 22.04 LTS,
so I downgraded the system. Since the tester is also the
host having connection issues, I will try downgrading that
system to 20.04, and see if that changes anything.<br>
</div>
</div>
</blockquote>
<br>
Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04,
Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38
every night.<br>
Right now Debian 11 is used for test engine in nightly
regressions.<br>
<br>
I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>I did try passing in the "--vg-rcf" argument to the
run.sh script of the test suite after installing valgrind,
but there was no additional output that I saw.<br>
</div>
</div>
</blockquote>
<br>
Sorry, I should valgrind output should be in valgrind.te_rcf
(direction where you run test engine).<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I will try pulling in the changes you've pushed up, and
will see if that fixes anything.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 25, 2023 at
9:57 AM Andrew Rybchenko <<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hello Adam, <br>
<br>
On 8/24/23 23:54, Andrew Rybchenko wrote:<br>
</div>
<blockquote type="cite">I'd like to try to repeat the
problem locally. Which Linux distro is running on test
engine and agents? <br>
<br>
In fact I know one problem with Debian 12 and Fedora 38
and we have <br>
patch in review to fix it, however, the behaviour is
different in <br>
this case, so it is unlike the same problem. <br>
</blockquote>
<br>
I've just published a new tag which fixes known test
engine side problems on Debian 12 and Fedora 38.<br>
<br>
<blockquote type="cite"> <br>
One more idea is to install valgrind on the test engine
host and <br>
run with option --vg-rcf to check if something weird is
happening. <br>
<br>
What I don't understand right now is why I see just one
failed attempt <br>
to connect in your log.txt and then Logger shutdown
after 9 minutes. <br>
<br>
Andrew. <br>
<br>
On 8/24/23 23:29, Adam Hassick wrote: <br>
<blockquote type="cite"> > Is there any firewall in
the network or on test hosts which could block
incoming TCP connection to the port 23571 <a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host where you run test engine? <br>
<br>
Our test engine host and the testbed are on the same
subnet. The connection does work sometimes. <br>
<br>
> If behaviour the same on the next try and you
see that test agent is kept running, could you check
using <br>
> <br>
> # netstat -tnlp <br>
> <br>
> that Test Agent is listening on the port and try
to establish TCP connection from test agent using <br>
> <br>
> $ telnet <a href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank">iol-dts-tester.dpdklab.iol.unh.edu</a>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571 <a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
<br>
> <br>
> and check if TCP connection could be
established. <br>
<br>
I was able to replicate the same behavior again, where
it hangs while RCF is trying to start. <br>
Running this command, I see this in the output: <br>
<br>
tcp 0 0 <a href="http://0.0.0.0:23571" target="_blank">0.0.0.0:23571</a>
<a href="http://0.0.0.0:23571" target="_blank"><http://0.0.0.0:23571></a>
0.0.0.0:* LISTEN 18599/ta <br>
<br>
So it seems like it is listening on the correct port.
<br>
Additionally, I was able to connect to the Tester
machine from our Test Engine host using telnet. It
printed the PID of the process once the connection was
opened. <br>
<br>
I tried running the "ta" application manually on the
command line, and it didn't print anything at all. <br>
Maybe the issue is something on the Test Engine side.
<br>
<br>
On Thu, Aug 24, 2023 at 2:35 PM Andrew Rybchenko <<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>
<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Hi Adam, <br>
<br>
> On the tester host (which appears to be the
Peer agent), there <br>
are four processes that I see running, which look
like the test <br>
agent processes. <br>
<br>
Before the next try I'd recommend to kill these
processes. <br>
<br>
Is there any firewall in the network or on test
hosts which could <br>
block incoming TCP connection to the port 23571 <br>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
from the host <br>
where you run test engine? <br>
<br>
If behaviour the same on the next try and you see
that test agent is <br>
kept running, could you check using <br>
<br>
# netstat -tnlp <br>
<br>
that Test Agent is listening on the port and try
to establish TCP <br>
connection from test agent using <br>
<br>
$ telnet <a href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank">iol-dts-tester.dpdklab.iol.unh.edu</a>
<br>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
23571 <br>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
<br>
<br>
and check if TCP connection could be established.
<br>
<br>
Another idea is to login Tester under root as
testing does, get <br>
start TA command from the log and try it by hands
without -n and <br>
remove extra escaping. <br>
<br>
# sudo
PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1 <br>
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=<br>
<br>
Hopefully in this case test agent directory
remains in the /tmp and <br>
you don't need to copy it as testing does. <br>
May be output could shed some light on what's
going on. <br>
<br>
Andrew. <br>
<br>
On 8/24/23 17:30, Adam Hassick wrote: <br>
<blockquote type="cite"> Hi Andrew, <br>
<br>
This is the output that I see in the terminal
when this failure <br>
occurs, after the test agent binaries build and
the test engine <br>
starts: <br>
<br>
Platform default build - pass <br>
Simple RCF consistency check succeeded <br>
--->>> Starting Logger...done <br>
--->>> Starting
RCF...rcf_net_engine_connect(): Connection timed <br>
out <a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank">iol-dts-tester.dpdklab.iol.unh.edu:23571</a>
<br>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target="_blank"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
<br>
<br>
Then, it hangs here until I kill the "te_rcf"
and "te_tee" <br>
processes. I let it hang for around 9 minutes. <br>
<br>
On the tester host (which appears to be the Peer
agent), there are <br>
four processes that I see running, which look
like the test agent <br>
processes. <br>
<br>
ta.Peer is an empty file. I've attached the
log.txt from this run. <br>
<br>
- Adam <br>
<br>
On Thu, Aug 24, 2023 at 4:22 AM Andrew Rybchenko
<br>
<<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>
<br>
<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Hi Adam, <br>
<br>
Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've
double-checked <br>
that it goes to 'copy_timeout' in
ts-conf/rcf.conf. <br>
Description in in
doc/sphinx/pages/group_te_engine_rcf.rst <br>
says that copy_timeout is in seconds and
implementation in <br>
lib/rcfunix/rcfunix.c passes the value to
select() tv_sec. <br>
Theoretically select() could be interrupted
by signal, but I <br>
think it is unlikely here. <br>
<br>
I'm not sure that I understand what do you
mean by RCF <br>
connection timeout. Does it happen on TE
startup when RCF <br>
starts test agents. If so,
TE_RCFUNIX_TIMEOUT could help. Or <br>
does it happen when tests are in progress,
e.g. in the middle <br>
of a test. If so, TE_RCFUNIX_TIMEOUT is
unrelated and most <br>
likely either host with test agent dies or
test agent itself <br>
crashes. It would be easier for me if
classify it if you share <br>
text log (log.txt, full or just
corresponding fragment with <br>
some context). Also content of ta.DPDK or
ta.Peer file <br>
depending on which agent has problems could
shed some light. <br>
Corresponding files contain stdout/stderr of
test agents. <br>
<br>
Andrew. <br>
<br>
On 8/23/23 17:45, Adam Hassick wrote: <br>
<blockquote type="cite"> Hi Andrew, <br>
<br>
I've set up a test rig repository here,
and have created <br>
configurations for our development testbed
based off of the <br>
examples. <br>
We've been able to get the test suite to
run manually on <br>
Mellanox CX5 devices once. <br>
However, we are running into an issue
where, when RCF starts, <br>
the RCF connection times out very
frequently. We aren't sure <br>
why this is the case. <br>
It works sometimes, but most of the time
when we try to run <br>
the test engine, it encounters this issue.
<br>
I've tried changing the RCF port by
setting <br>
"TE_RCF_PORT=<some port number>" and
rebooting the testbed <br>
machines. Neither seems to fix the issue.
<br>
<br>
It also seems like the timeout takes far
longer than 60 <br>
seconds, even when running "export
TE_RCFUNIX_TIMEOUT=60" <br>
before I try to run the test suite. <br>
I assume the unit for this variable is
seconds? <br>
<br>
Thanks, <br>
Adam <br>
<br>
On Mon, Aug 21, 2023 at 10:19 AM Adam
Hassick <br>
<<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>>
wrote: <br>
<br>
Hi Andrew, <br>
<br>
Thanks, I've cloned the example
repository and will start <br>
setting up a configuration for our
development testbed <br>
today. I'll let you know if I run into
any difficulties <br>
or have any questions. <br>
<br>
- Adam <br>
<br>
On Sun, Aug 20, 2023 at 4:40 AM Andrew
Rybchenko <br>
<<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>
<br>
<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Hi Adam, <br>
<br>
I've published <br>
<a href="https://github.com/ts-factory/ts-rigs-sample" target="_blank">https://github.com/ts-factory/ts-rigs-sample</a>
<br>
<a href="https://github.com/ts-factory/ts-rigs-sample" target="_blank"><https://github.com/ts-factory/ts-rigs-sample></a>.
<br>
Hopefully it will help to define
your test rigs and <br>
successfully run some tests
manually. Feel free to <br>
ask any questions and I'll answer
here and try to <br>
update documentation. <br>
<br>
Meanwhile I'll prepare missing
bits for steps (2) and <br>
(3). <br>
Hopefully everything is in place
for step (4), but we <br>
need to make steps (2) and (3)
first. <br>
<br>
Andrew. <br>
<br>
On 8/18/23 21:40, Andrew Rybchenko
wrote: <br>
<blockquote type="cite"> Hi Adam, <br>
<br>
> I've conferred with the
rest of the team, and we <br>
think it would be best to move
forward with mainly <br>
option B. <br>
<br>
OK, I'll provide the sample on
Monday for you. It is <br>
almost ready right now, but I
need to double-check <br>
it before publishing. <br>
<br>
Regards, <br>
Andrew. <br>
<br>
On 8/17/23 20:03, Adam Hassick
wrote: <br>
<blockquote type="cite"> Hi
Andrew, <br>
<br>
I'm adding the CI mailing list
to this <br>
conversation. Others in the
community might find <br>
this conversation valuable. <br>
<br>
We do want to run testing on a
regular basis. The <br>
Jenkins integration will be
very useful for us, as <br>
most of our CI is orchestrated
by Jenkins. <br>
I've conferred with the rest
of the team, and we <br>
think it would be best to move
forward with mainly <br>
option B. <br>
If you would like to know
anything about our <br>
testbeds that would help you
with creating an <br>
example ts-rigs repo, I'd be
happy to answer any <br>
questions you have. <br>
<br>
We have multiple test rigs (we
call these <br>
"DUT-tester pairs") that we
run our existing <br>
hardware testing on, with
differing network <br>
hardware and CPU architecture.
I figured this might <br>
be an important detail. <br>
<br>
Thanks, <br>
Adam <br>
<br>
On Thu, Aug 17, 2023 at
11:44 AM Andrew Rybchenko <br>
<<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank">andrew.rybchenko@oktetlabs.ru</a>
<br>
<a href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
wrote: <br>
<br>
Greatings Adam, <br>
<br>
I'm happy to hear that
you're trying to bring <br>
it up. <br>
<br>
As I understand the final
goal is to run it on <br>
regular basis. So, we need
to make it properly <br>
from the very beginning. <br>
Bring up of all features
consists of 4 steps: <br>
<br>
1. Create site-specific
repository (we call it <br>
ts-rigs) which contains
information about test <br>
rigs and other
site-specific information like <br>
where to send mails, where
to store logs etc. <br>
It is required for manual
execution as well, <br>
since test rigs
description is essential. I'll <br>
return to the topic below.
<br>
<br>
2. Setup logs storage for
automated runs. <br>
Basically it is a disk
space plus apache2 web <br>
server with few CGI
scripts which help a lot to <br>
save disk space. <br>
<br>
3. Setup Bublik web
application which provides <br>
web interface to view
testing results. Same as <br>
<a href="https://ts-factory.io/bublik" target="_blank">https://ts-factory.io/bublik</a>
<br>
<a href="https://ts-factory.io/bublik" target="_blank"><https://ts-factory.io/bublik></a>
<br>
<br>
4. Setup Jenkins to run
tests on regularly, <br>
save logs in log storage
(2) and import it to <br>
bublik (3). <br>
<br>
Last few month we spent on
our homework to make <br>
it simpler to bring up
automated execution <br>
using Jenkins - <br>
<a href="https://github.com/ts-factory/te-jenkins" target="_blank">https://github.com/ts-factory/te-jenkins</a>
<br>
<a href="https://github.com/ts-factory/te-jenkins" target="_blank"><https://github.com/ts-factory/te-jenkins></a>
<br>
Corresponding bits in
dpdk-ethdev-ts will be <br>
available tomorrow. <br>
<br>
Let's return to the step
(1). <br>
<br>
Unfortunately there is no
publicly available <br>
example of the ts-rigs
repository since <br>
sensitive site-specific
information is located <br>
there. But I'm ready to
help you to create it <br>
for UNH. I see two options
here: <br>
<br>
(A) I'll ask questions and
based on your <br>
answers will create the
first draft with my <br>
comments. <br>
<br>
(B) I'll make a
template/example ts-rigs repo, <br>
publish it and you'll
create UNH ts-rigs based <br>
on it. <br>
<br>
Of course, I'll help to
debug and finally bring <br>
it up in any case. <br>
<br>
(A) is a bit simpler for
me and you, but (B) is <br>
a bit more generic and
will help other <br>
potential users to bring
it up. <br>
We can combine (A)+(B).
I.e. start from (A). <br>
What do you think? <br>
<br>
Thanks, <br>
Andrew. <br>
<br>
On 8/17/23 15:18,
Konstantin Ushakov wrote: <br>
<blockquote type="cite">
Greetings Adam, <br>
<br>
<br>
Thanks for contacting
us. I copy Andrew who <br>
would be happy to help <br>
<br>
Thanks, <br>
Konstantin <br>
<br>
<blockquote type="cite">
On 16 Aug 2023, at 21:50, Adam Hassick <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><ahassick@iol.unh.edu></a>
<br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
wrote: <br>
<br>
<br>
Greetings Konstantin,
<br>
<br>
I am in the process of
setting up the DPDK <br>
Poll Mode Driver test
suite as an addition to <br>
our testing coverage
for DPDK at the UNH lab. <br>
<br>
I have some questions
about how to set the <br>
test suite arguments.
<br>
<br>
I have been able to
configure the Test Engine <br>
to connect to the
hosts in the testbed. The <br>
RCF, Configurator, and
Tester all begin to <br>
run, however the
prelude of the test suite <br>
fails to run. <br>
<br>
<a href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters" target="_blank">https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters</a>
<a href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters" target="_blank"><https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters></a>
<br>
<br>
The documentation
mentions that there are <br>
several test
parameters for the test suite, <br>
like for the IUT test
link MAC, etc. These <br>
seem like they would
need to be set somewhere <br>
to run many of the
tests. <br>
<br>
I see in the Test
Engine documentation, there <br>
are instructions on
how to create new <br>
parameters for test
suites in the Tester <br>
configuration, but
there is nothing in the <br>
user guide or in the
Tester guide for how to <br>
set the arguments for
the parameters when <br>
running the test suite
that I can find. I'm <br>
not sure if I need to
write my own Tester <br>
config, or if I should
be setting these in <br>
some other way. <br>
<br>
How should these
values be set? <br>
<br>
I'm also not sure what
environment <br>
variables/arguments
are strictly necessary or <br>
which are optional. <br>
<br>
Regards, <br>
Adam <br>
<br>
--
*Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability
Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
</blockquote>
<br>
<br>
<br>
-- *Adam
Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
<br>
</blockquote>
<br>
<br>
<br>
-- *Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
<br>
<br>
<br>
-- *Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a>
<a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
<br>
<br>
<br>
-- *Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a> <a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
<br>
<br>
<br>
-- <br>
*Adam Hassick* <br>
Senior Developer <br>
UNH InterOperability Lab <br>
<a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a>
<a href="mailto:ahassick@iol.unh.edu" target="_blank"><mailto:ahassick@iol.unh.edu></a>
<br>
<a href="http://iol.unh.edu" target="_blank">iol.unh.edu</a> <a href="https://www.iol.unh.edu/" target="_blank"><https://www.iol.unh.edu/></a>
<br>
+1 (603) 475-8248 <br>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span style="background-color:rgb(255,255,255)"><span style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/" target="_blank">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><b><span style="background-color:rgb(255,255,255)"><span style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br></div><span style="color:rgb(102,102,102)"></span></div><div><span style="color:rgb(102,102,102)">Senior Developer</span></div><div><span style="color:rgb(102,102,102)"><span style="color:rgb(11,83,148)"><span style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span><span style="color:rgb(102,102,102)"></span></div><div><span style="color:rgb(102,102,102)"><a href="mailto:ahassick@iol.unh.edu" target="_blank">ahassick@iol.unh.edu</a><br></span></div><div><span style="color:rgb(102,102,102)"><a href="https://www.iol.unh.edu/" target="_blank">iol.unh.edu</a><br></span></div>+1 (603) 475-8248<br></div></div>