<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hi Adam,<br>
<br>
> On the tester host (which appears to be the Peer agent),
there are four processes that I see running, which look like the
test agent processes.<br>
<br>
Before the next try I'd recommend to kill these processes. <br>
<br>
Is there any firewall in the network or on test hosts which could
block incoming TCP connection to the port <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571">23571</a>
from the host where you run test engine?<br>
<br>
If behaviour the same on the next try and you see that test agent
is kept running, could you check using<br>
<br>
# netstat -tnlp<br>
<br>
that Test Agent is listening on the port and try to establish TCP
connection from test agent using<br>
<br>
$ telnet <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571">iol-dts-tester.dpdklab.iol.unh.edu</a>
<a href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571">23571</a><br>
<br>
and check if TCP connection could be established.<br>
<br>
Another idea is to login Tester under root as testing does, get
start TA command from the log and try it by hands without -n and
remove extra escaping.<br>
<br>
# sudo PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=<br>
<br>
Hopefully in this case test agent directory remains in the /tmp
and you don't need to copy it as testing does.<br>
May be output could shed some light on what's going on.<br>
<br>
Andrew.<br>
<br>
On 8/24/23 17:30, Adam Hassick wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAC-YWqgvgKRgffwWY3mqWbC8o-LJ_o0BkRuuBxDSqg8Pnj0h1Q@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>
<div>Hi Andrew,<br>
<br>
</div>
<div>This is the output that I see in the terminal when this
failure occurs, after the test agent binaries build and the
test engine starts:<br>
</div>
<div><br>
Platform default build - pass<br>
Simple RCF consistency check succeeded<br>
--->>> Starting Logger...done<br>
--->>> Starting RCF...rcf_net_engine_connect():
Connection timed out <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu:23571</a><br>
<br>
</div>
<div>Then, it hangs here until I kill the "te_rcf" and
"te_tee" processes. I let it hang for around 9 minutes.<br>
<br>
</div>
<div>On the tester host (which appears to be the Peer agent),
there are four processes that I see running, which look like
the test agent processes.<br>
</div>
<div><br>
</div>
<div>ta.Peer is an empty file. I've attached the log.txt from
this run.<br>
<br>
</div>
<div> - Adam<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 24, 2023 at
4:22 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
that it goes to 'copy_timeout' in ts-conf/rcf.conf.<br>
Description in in doc/sphinx/pages/group_te_engine_rcf.rst
says that copy_timeout is in seconds and implementation in
lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
Theoretically select() could be interrupted by signal, but
I think it is unlikely here.<br>
<br>
I'm not sure that I understand what do you mean by RCF
connection timeout. Does it happen on TE startup when RCF
starts test agents. If so, TE_RCFUNIX_TIMEOUT could help.
Or does it happen when tests are in progress, e.g. in the
middle of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated
and most likely either host with test agent dies or test
agent itself crashes. It would be easier for me if
classify it if you share text log (log.txt, full or just
corresponding fragment with some context). Also content of
ta.DPDK or ta.Peer file depending on which agent has
problems could shed some light. Corresponding files
contain stdout/stderr of test agents.<br>
<br>
Andrew.<br>
<br>
On 8/23/23 17:45, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi Andrew,<br>
<br>
</div>
I've set up a test rig repository here, and
have created configurations for our
development testbed based off of the examples.<br>
We've been able to get the test suite to run
manually on Mellanox CX5 devices once.<br>
However, we are running into an issue where,
when RCF starts, the RCF connection times out
very frequently. We aren't sure why this is
the case.<br>
</div>
It works sometimes, but most of the time when we
try to run the test engine, it encounters this
issue.<br>
</div>
<div>I've tried changing the RCF port by setting
"TE_RCF_PORT=<some port number>" and
rebooting the testbed machines. Neither seems to
fix the issue.<br>
</div>
<div><br>
</div>
It also seems like the timeout takes far longer
than 60 seconds, even when running "export
TE_RCFUNIX_TIMEOUT=60" before I try to run the
test suite.<br>
</div>
I assume the unit for this variable is seconds?<br>
<br>
</div>
Thanks,<br>
</div>
Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Aug 21, 2023
at 10:19 AM Adam Hassick <<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div>
<div>Hi Andrew,<br>
<br>
</div>
Thanks, I've cloned the example repository and
will start setting up a configuration for our
development testbed today. I'll let you know if
I run into any difficulties or have any
questions.<br>
<br>
</div>
- Adam<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Aug 20,
2023 at 4:40 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Adam,<br>
<br>
I've published <a
href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/ts-factory/ts-rigs-sample</a>.
Hopefully it will help to define your test
rigs and successfully run some tests
manually. Feel free to ask any questions and
I'll answer here and try to update
documentation.<br>
<br>
Meanwhile I'll prepare missing bits for
steps (2) and (3).<br>
Hopefully everything is in place for step
(4), but we need to make steps (2) and (3)
first.<br>
<br>
Andrew.<br>
<br>
On 8/18/23 21:40, Andrew Rybchenko wrote:<br>
</div>
<blockquote type="cite">
<div>Hi Adam,<br>
<br>
> I've conferred with the rest of the
team, and we think it would be best to
move forward with mainly option B.<br>
<br>
OK, I'll provide the sample on Monday for
you. It is almost ready right now, but I
need to double-check it before publishing.<br>
<br>
Regards,<br>
Andrew.<br>
<br>
On 8/17/23 20:03, Adam Hassick wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Andrew,<br>
<br>
</div>
<div>I'm adding the CI mailing list to
this conversation. Others in the
community might find this
conversation valuable.<br>
</div>
<div><br>
</div>
We do want to run testing on a regular
basis. The Jenkins integration will be
very useful for us, as most of our CI
is orchestrated by Jenkins.<br>
</div>
<div>I've conferred with the rest of the
team, and we think it would be best to
move forward with mainly option B.<br>
</div>
<div>If you would like to know anything
about our testbeds that would help you
with creating an example ts-rigs repo,
I'd be happy to answer any questions
you have.<br>
</div>
<div><br>
</div>
<div>We have multiple test rigs (we call
these "DUT-tester pairs") that we run
our existing hardware testing on, with
differing network hardware and CPU
architecture. I figured this might be
an important detail.<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Adam<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Thu, Aug 17, 2023 at 11:44 AM Andrew
Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div>Greatings Adam,<br>
<br>
I'm happy to hear that you're
trying to bring it up.<br>
<br>
As I understand the final goal is
to run it on regular basis. So, we
need to make it properly from the
very beginning.<br>
Bring up of all features consists
of 4 steps:<br>
<br>
1. Create site-specific repository
(we call it ts-rigs) which
contains information about test
rigs and other site-specific
information like where to send
mails, where to store logs etc. It
is required for manual execution
as well, since test rigs
description is essential. I'll
return to the topic below.<br>
<br>
2. Setup logs storage for
automated runs. Basically it is a
disk space plus apache2 web server
with few CGI scripts which help a
lot to save disk space.<br>
<br>
3. Setup Bublik web application
which provides web interface to
view testing results. Same as <a
href="https://ts-factory.io/bublik" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://ts-factory.io/bublik</a><br>
<br>
4. Setup Jenkins to run tests on
regularly, save logs in log
storage (2) and import it to
bublik (3).<br>
<br>
Last few month we spent on our
homework to make it simpler to
bring up automated execution using
Jenkins - <a
href="https://github.com/ts-factory/te-jenkins"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/ts-factory/te-jenkins</a><br>
Corresponding bits in
dpdk-ethdev-ts will be available
tomorrow.<br>
<br>
Let's return to the step (1).<br>
<br>
Unfortunately there is no publicly
available example of the ts-rigs
repository since sensitive
site-specific information is
located there. But I'm ready to
help you to create it for UNH. I
see two options here:<br>
<br>
(A) I'll ask questions and based
on your answers will create the
first draft with my comments.<br>
<br>
(B) I'll make a template/example
ts-rigs repo, publish it and
you'll create UNH ts-rigs based on
it.<br>
<br>
Of course, I'll help to debug and
finally bring it up in any case.<br>
<br>
(A) is a bit simpler for me and
you, but (B) is a bit more generic
and will help other potential
users to bring it up.<br>
We can combine (A)+(B). I.e. start
from (A). What do you think?<br>
<br>
Thanks,<br>
Andrew.<br>
<br>
On 8/17/23 15:18, Konstantin
Ushakov wrote:<br>
</div>
<blockquote type="cite"> Greetings
Adam,
<div><br>
</div>
<div><br>
</div>
<div>Thanks for contacting us. I
copy Andrew who would be happy
to help<br>
<br>
<div dir="ltr">Thanks,
<div>Konstantin</div>
</div>
<div dir="ltr"><br>
<blockquote type="cite">On 16
Aug 2023, at 21:50, Adam
Hassick <a
href="mailto:ahassick@iol.unh.edu"
target="_blank"
moz-do-not-send="true"><ahassick@iol.unh.edu></a>
wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>
<div>
<div>
<div>Greetings
Konstantin,<br>
<br>
</div>
I am in the process
of setting up the
DPDK Poll Mode
Driver test suite as
an addition to our
testing coverage for
DPDK at the UNH lab.<br>
<br>
</div>
I have some questions
about how to set the
test suite arguments.<br>
<br>
</div>
<div>I have been able to
configure the Test
Engine to connect to
the hosts in the
testbed. The RCF,
Configurator, and
Tester all begin to
run, however the
prelude of the test
suite fails to run.<br>
</div>
<div><br>
<a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters</a><br>
<br>
</div>
The documentation
mentions that there are
several test parameters
for the test suite, like
for the IUT test link
MAC, etc. These seem
like they would need to
be set somewhere to run
many of the tests.<br>
<br>
I see in the Test Engine
documentation, there are
instructions on how to
create new parameters
for test suites in the
Tester configuration,
but there is nothing in
the user guide or in the
Tester guide for how to
set the arguments for
the parameters when
running the test suite
that I can find. I'm not
sure if I need to write
my own Tester config, or
if I should be setting
these in some other way.<br>
<br>
How should these values
be set?<br>
<br>
</div>
<div>I'm also not sure
what environment
variables/arguments are
strictly necessary or
which are optional.<br>
</div>
<div><br>
</div>
<div>Regards,<br>
</div>
<div>Adam<br>
</div>
<div>
<div>
<div>
<div>
<div>
<div><br>
<span
class="gmail_signature_prefix">--
</span><br>
<div dir="ltr"
class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam
Hassick</span></span></b><br>
</div>
<span
style="color:rgb(102,102,102)"></span></div>
<div><span
style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span
style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603)
475-8248<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span
style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span
style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span
style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span
style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/"
target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam
Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span
style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior
Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div><b><span style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
</div>
<span style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)">Senior Developer</span></div>
<div><span style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
<div><span style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
</span></div>
<div><span style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank"
moz-do-not-send="true">iol.unh.edu</a><br>
</span></div>
+1 (603) 475-8248<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>