<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">Hi Adam,<br>
      <br>
      I've pushed new TE tag v1.20.0 which supported a new command-line
      option --tester-dial=NUM where NUM is from 0 to 100. it allows to
      choose percentage of tests to run. If you want stable set, you
      should pass --tester-random-seed=0 (or other integer). It is the
      first sketch and we have plans to improve it, but feedback would
      be welcome.<br>
      <br>
      > Is it needed on the tester?<br>
      <br>
      It is hard to say if it is strictly required for simple tests.
      However, it is better to update Tester as well, since performance
      tests run DPDK on Tester as well.<br>
      <br>
      > Are there any other manual setup steps for these devices that
      I might be missing?<br>
      <br>
      I don't remember anything else.<br>
      <br>
      I think it is better to get down to details and take a look at
      logs. I'm ready to help with it and explain what's happening
      there. May be it will help to understand if it is a problem with
      setup/configuration.<br>
      <br>
      Text logs are not very convenient. Ideally logs should be imported
      to bublik, however, manual runs do not provide all required
      artifacts right now (Jenkins jobs generate all required
      artifacts).<br>
      Other option is 'tmp_raw_log' file (should be packed to make it
      smaller) which could be converted to various log formats.<br>
      Would it be OK for you if I import your logs to bublik at
      ts-factory.io? Or is it a problem that it is publicly available?<br>
      Would it help if we add authentication and access control there?<br>
      <br>
      Andrew.<br>
      <br>
      On 9/8/23 17:57, Adam Hassick wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAC-YWqjOhiYXkBAK5pLnMqUNkB-z4Y=JAUMeN3fc6AMfjOzPrg@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div>
          <div>
            <div>
              <div>
                <div>Hi Andrew,<br>
                  <br>
                </div>
                I have a couple questions about needed setup of the NICs
                for the ethdev test suite.<br>
                <br>
              </div>
              Our MCX5s and XL710s are failing the checkup tests. The
              pass rate appears to be much worse on the XL710s (40 of 73
              tests failed, 3 passed unexpectedly).<br>
              <br>
            </div>
            For the XL710s, I've updated the driver and NVM versions to
            match the minimum supported versions in the compatibility
            matrix found on the DPDK documentation. This did not change
            the failure rate much.<br>
          </div>
          For the MCX5s, I've installed the latest LTS version of the
          OFED bifurcated driver on the DUT. Is it needed on the tester?<br>
          <br>
        </div>
        Are there any other manual setup steps for these devices that I
        might be missing?<br>
        <div>
          <div>
            <div>
              <div>
                <div>
                  <div>
                    <div><br>
                    </div>
                    <div>Thanks,<br>
                    </div>
                    <div>Adam<br>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Wed, Sep 6, 2023 at
          11:00 AM Adam Hassick <<a
            href="mailto:ahassick@iol.unh.edu" moz-do-not-send="true"
            class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div>
              <div>
                <div>
                  <div>
                    <div>Hi Andrew,<br>
                      <br>
                    </div>
                    <div>Yes, I copied the X710 configs to set up XL710
                      configs. I changed the environment variable names
                      from the X710 suffix to XL710 suffix in the
                      script, and forgot to change them in the
                      corresponding environment file.<br>
                    </div>
                  </div>
                  That fixed the issue.<br>
                  <br>
                </div>
                I got the checkup tests working on the XL710 now. Most
                of them are failing, which leads me to believe this is
                an issue with our testbed. Based on the DPDK
                documentation for i40e, the firmware and driver versions
                are much older than what DPDK 22.11 LTS and main prefer,
                so I'll try updating those.<br>
                <br>
              </div>
              For now I'm working on getting the XL710 checkup tests
              passing, and will pick up getting the E810 configured
              properly next. I'll let you know if I run into any more
              issues in relation to the test engine.<br>
              <br>
            </div>
            <div>Thanks,<br>
            </div>
            <div>Adam<br>
            </div>
          </div>
          <br>
          <div class="gmail_quote">
            <div dir="ltr" class="gmail_attr">On Wed, Sep 6, 2023 at
              7:36 AM Andrew Rybchenko <<a
                href="mailto:andrew.rybchenko@oktetlabs.ru"
                target="_blank" moz-do-not-send="true"
                class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
              wrote:<br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div>
                <div>Hi Adam,<br>
                  <br>
                  On 9/5/23 18:01, Adam Hassick wrote:<br>
                </div>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div>
                      <div>
                        <div>
                          <div>
                            <div>
                              <div>Hi Andrew,<br>
                                <br>
                              </div>
                              The compilation warning issue is now
                              resolved. Again, thank you guys for fixing
                              this for us. I can run the tests on the
                              Mellanox CX5s again, however I'm running
                              into a couple new issues with running the
                              prologues on the Intel cards.<br>
                              <br>
                            </div>
                            When running testing on the Intel XL710s, I
                            see this error appear in the log:<br>
                            <br>
                            <blockquote class="gmail_quote"
                              style="margin:0px 0px 0px
                              0.8ex;border-left:1px solid
                              rgb(204,204,204);padding-left:1ex">ERROR
                               prologue  Environment LIB  14:16:13.650<br>
                              Too few networks in available
                              configuration (0) in comparison with
                              required (1)<br>
                            </blockquote>
                            <br>
                          </div>
                          This seems like a trivial configuration error,
                          perhaps this is something I need to set up in
                          ts-rigs. I briefly searched through the
                          examples there and didn't see any mention of
                          how to set up a network.<br>
                        </div>
                        <div>I will attach this log just in case you
                          need more information.<br>
                        </div>
                      </div>
                    </div>
                  </div>
                </blockquote>
                <br>
                Unfortunately logs are insufficient to understand it.
                I've pushed new tag to TE v1.19.0 which add log message
                with TE_* environment variables.<br>
                Most likely something is wrong with variables which are
                used as conditions when available networks are defined
                in ts-conf/cs/inc.net_cfg_pci_fns.yml:<br>
                TE_PCI_INSTANCE_IUT_TST1<br>
                TE_PCI_INSTANCE_IUT_TST1a<br>
                TE_PCI_INSTANCE_TST1a_IUT<br>
                TE_PCI_INSTANCE_TST1_IUT<br>
                My guess it that you change naming a bit, but script
                like ts-rigs-sample/scripts/iut.h1-x710 is not included
                or not updated.<br>
                <br>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div>
                      <div>There is a different error when running on
                        the Intel E810s. It appears to me like it starts
                        DPDK, does some configuration inside DPDK and on
                        the device, and then fails to bring the device
                        back up. Since this error seems very
                        non-trivial, I will also attach this log.<br>
                      </div>
                    </div>
                  </div>
                </blockquote>
                <br>
                This one is a bit simpler. Few lines after the first
                ERROR in log I see the following:<br>
                WARN  RCF  DPDK  13:06:00.144<br>
                ice_program_hw_rx_queue(): currently package doesn't
                support RXDID (22)<br>
                ice_rx_queue_start(): fail to program RX queue 0<br>
                ice_dev_start(): fail to start Rx queue 0<br>
                Device with port_id=0 already stopped<br>
                <br>
                It is stdout/stderr from test agent which runs DPDK.
                Same logs in plain format are available in ta.DPDK file.<br>
                I'm not an expert here, but I vaguely remember that E810
                requires correct firmware and DDP to be loaded.<br>
                There is some information in
                dpdk/doc/guides/nics/ice.rst.<br>
                <br>
                You can try to add --dev-args=safe-mode-support=1
                command-line option described there.<br>
                <br>
                Hope it helps,<br>
                Andrew.<br>
                <br>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div>
                      <div><br>
                      </div>
                      Thanks,<br>
                    </div>
                    Adam<br>
                  </div>
                  <br>
                  <div class="gmail_quote">
                    <div dir="ltr" class="gmail_attr">On Fri, Sep 1,
                      2023 at 3:59 AM Andrew Rybchenko <<a
                        href="mailto:andrew.rybchenko@oktetlabs.ru"
                        target="_blank" moz-do-not-send="true"
                        class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
                      wrote:<br>
                    </div>
                    <blockquote class="gmail_quote" style="margin:0px
                      0px 0px 0.8ex;border-left:1px solid
                      rgb(204,204,204);padding-left:1ex">
                      <div>
                        <div>Hi Adam,<br>
                          <br>
                          On 8/31/23 22:38, Adam Hassick wrote:<br>
                        </div>
                        <blockquote type="cite">
                          <div dir="ltr">
                            <div>Hi Andrew,<br>
                            </div>
                            <div><br>
                              I have one additional question as well:
                              Does the test engine support running tests
                              on two ARMv8 test agents?</div>
                            <div><br>
                            </div>
                            <div>
                              <blockquote class="gmail_quote"
                                style="margin:0px 0px 0px
                                0.8ex;border-left:1px solid
                                rgb(204,204,204);padding-left:1ex">1.
                                We'll sort out warnings this week.
                                Thanks for heads up.<br>
                              </blockquote>
                              <div><br>
                              </div>
                              <div>Great. Let me know when that's fixed.</div>
                            </div>
                          </div>
                        </blockquote>
                        <br>
                        Done. We also fixed a number of warnings in TE.<br>
                        Also we fixed root test package name to be
                        consistent with the repository name.<br>
                        <br>
                        <blockquote type="cite">
                          <div dir="ltr">
                            <blockquote class="gmail_quote"
                              style="margin:0px 0px 0px
                              0.8ex;border-left:1px solid
                              rgb(204,204,204);padding-left:1ex">
                              <div>Support for old LTS branches was
                                dropped some time ago, but in the future
                                it is definitely possible to keep it for
                                new LTS branches. I think 22.11 is
                                supported, but I'm not sure about older
                                LTS releases.</div>
                            </blockquote>
                            <div><br>
                            </div>
                            <div>Good to know.<br>
                              <div> <br>
                                <blockquote class="gmail_quote"
                                  style="margin:0px 0px 0px
                                  0.8ex;border-left:1px solid
                                  rgb(204,204,204);padding-left:1ex"> 2.
                                  You can add command-line option
                                  --sanity to run tests marked with
                                  TEST_HARNESS_SANITY requirement (see
                                  dpdk-ethdev-ts/scripts/run.sh and grep
                                  TEST_HARNESS_SANITY dpdk-ethdev-ts to
                                  see which tests are marked). Yes,
                                  there is a space for terminology
                                  improvement here. We'll do it.<br>
                                </blockquote>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                        <br>
                        Done. Now it is called --checkup.<br>
                        <br>
                        <blockquote type="cite">
                          <div dir="ltr">
                            <div>
                              <div>
                                <blockquote class="gmail_quote"
                                  style="margin:0px 0px 0px
                                  0.8ex;border-left:1px solid
                                  rgb(204,204,204);padding-left:1ex"><br>
                                  Also it takes a lot of time because of
                                  failures and tests which wait for some
                                  timeout.<br>
                                </blockquote>
                              </div>
                              <div><br>
                              </div>
                              <div>That makes sense to me. We'll use the
                                time to complete tests on virtio or the
                                Intel devices as a reference for how
                                long the tests really take to complete.<br>
                              </div>
                              <div>We will explore the possibility of
                                periodically running the sanity tests
                                for patches.<br>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                        <br>
                        I'll double-check and let you know how long
                        entire TS runs on Intel X710, E810, Mellanox CX5
                        and virtio net. Just to ensure that time
                        observed in your case looks the same.<br>
                        <br>
                        <blockquote type="cite">
                          <div dir="ltr">
                            <div>
                              <div> <br>
                                <blockquote class="gmail_quote"
                                  style="margin:0px 0px 0px
                                  0.8ex;border-left:1px solid
                                  rgb(204,204,204);padding-left:1ex">
                                  The test harness can provide coverage
                                  reports based on gcov, but I'm not
                                  sure what you mean by a "dial" to
                                  control test coverage. Provided
                                  reports are rather for human to
                                  analyze.<br>
                                </blockquote>
                              </div>
                              <div><br>
                              </div>
                              <div>The general idea is to have some kind
                                of parameter on the test suite, which
                                could be an integer ranging from zero to
                                ten, that controls how many tests are
                                run based on how important the test is.<br>
                                <br>
                              </div>
                              <div>Similar to how some command line
                                interfaces provide a verbosity level
                                parameter (some number of "-v"
                                arguments) to control the importance of
                                the information in the log.<br>
                              </div>
                              The verbosity level zero only prints very
                              important log messages, while ten prints
                              everything.<br>
                            </div>
                            <div><br>
                              In much the same manner as above, this
                              "dial" parameter controls what tests are
                              run and with what parameters based on how
                              important those tests and test parameter
                              combinations are.<br>
                              Coverage Level zero tells the suite to run
                              a very basic set of important tests, with
                              minimal parameterization. This mode would
                              take only ~5-10 minutes to run.<br>
                              In contrast, Coverage Level ten includes
                              all the edge cases, every combination of
                              test parameters, everything the test suite
                              can do, which takes the normal several
                              hours to run.<br>
                              The values 1 - 9 are between those two
                              extremes, allowing the user to get a
                              gradient of test coverage in the results
                              and to limit the running time.<br>
                              <br>
                            </div>
                            Then we could, for example, run the "run.sh"
                            with a level of 2 or 3 for incoming patches
                            that need quick results, and with a level of
                            10 for the less often run periodic tests
                            performed on main or LTS branches.<br>
                          </div>
                        </blockquote>
                        <br>
                        Understood now. Thanks a lot for the idea. We'll
                        discuss it and come back.<br>
                        <br>
                        <blockquote type="cite">
                          <div dir="ltr">
                            <div>
                              <div>
                                <div> </div>
                                <blockquote class="gmail_quote"
                                  style="margin:0px 0px 0px
                                  0.8ex;border-left:1px solid
                                  rgb(204,204,204);padding-left:1ex">
                                  <div>
                                    <div> 3. Yes, really many tests on
                                      Mellanox CX5 NICs report
                                      unexpected testing results.
                                      Unfortunately it is time consuming
                                      to fill in expectations database
                                      since it is necessary to analyze
                                      testing results and classify if it
                                      is a bug or just acceptable
                                      behaviour aspect.<br>
                                      <br>
                                      Bublik allows to compare results
                                      of two runs. It is useful for
                                      human, but still not good for
                                      automation.<br>
                                      <br>
                                      I have local patch for mlx5 driver
                                      which reports Tx ring size
                                      maximum. It makes pass rate
                                      higher. It is a problem for test
                                      harness that mlx5 does not report
                                      limits right now.<br>
                                      <br>
                                      Pass rate on Intel X710 is about
                                      92% on my test rig. Pass rate on
                                      virtio net is 99% right now and
                                      could be done 100% easily (just
                                      one thing to fix in expectations).<br>
                                      <br>
                                      I think logs storage setup is
                                      essential for logs analysis. Of
                                      course, you can request HTML logs
                                      when you run tests
                                      (--log-html=html) or generate
                                      after run using
                                      dpdk-ethdev-ts/scripts/html-log.sh
                                      and open index.html in a browser,
                                      but logs storage makes it more
                                      convenient.<br>
                                    </div>
                                  </div>
                                </blockquote>
                                <div><br>
                                  We are interested in setting up
                                  Bublik, potentially as an
                                  externally-facing component, once we
                                  have our process of running the test
                                  suite stabilized.</div>
                                <div>Once we are able to run the test
                                  suite again, I'll see what the pass
                                  rate is on our other hardware.<br>
                                  Good to know that it isn't an issue
                                  with our dev testbed causing the high
                                  fail rate.</div>
                              </div>
                              <div>
                                <div><br>
                                </div>
                                <div>For Intel hardware, we have an
                                  XL710 and an Intel E810-C in our
                                  development testbed. Although they are
                                  slightly different devices, ideally
                                  the pass rate will be identical or
                                  similar. I have yet to set up a VM
                                  pair for virtio, but we will soon.<br>
                                </div>
                                <div><br>
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex">
                                    Latest version of test-environment
                                    has examples of our CGI scripts
                                    which we use for log storage (see
                                    tools/log_server/README.md).<br>
                                    <br>
                                    Also all bits for Jenkins setup are
                                    available. See
                                    dpdk-ethdev-ts/jenkins/README.md and
                                    examples of jenkins files in
                                    ts-rigs-sample.<br>
                                  </blockquote>
                                </div>
                                <div><br>
                                </div>
                                <div>Jenkins integration, setting up
                                  production rig configurations, and
                                  permanent log storage will be our next
                                  steps once I am able to run the tests
                                  again.<br>
                                </div>
                                <div>Unless there is an easy way to have
                                  meson not pass "-Werror" into GCC.
                                  Then I would be able to run the test
                                  suite.<br>
                                </div>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                        <br>
                        Hopefully it is resolved now.<br>
                        <br>
                        I thought a bit more about your usecase for
                        Jenkins. I'm not 100% sure that existing
                        pipelines are convenient for your usecase.<br>
                        Fill free to ask questions when you are on it.<br>
                        <br>
                        Thanks,<br>
                        Andrew.<br>
                        <br>
                        <blockquote type="cite">
                          <div dir="ltr">
                            <div>
                              <div>
                                <div><br>
                                </div>
                                <div>Thanks,<br>
                                </div>
                                <div>Adam<br>
                                </div>
                                <div><br>
                                </div>
                                <div> </div>
                                <blockquote class="gmail_quote"
                                  style="margin:0px 0px 0px
                                  0.8ex;border-left:1px solid
                                  rgb(204,204,204);padding-left:1ex">
                                  <div>
                                    <div> <br>
                                      On 8/29/23 17:02, Adam Hassick
                                      wrote:<br>
                                    </div>
                                    <blockquote type="cite">
                                      <div dir="ltr">
                                        <div>
                                          <div>
                                            <div>Hi Andrew,<br>
                                              <br>
                                            </div>
                                            That fix seems to have
                                            resolved the issue, thanks
                                            for the quick turnaround
                                            time on that patch.<br>
                                          </div>
                                          <div>Now that we have the RCF
                                            timeout issue resolved,
                                            there are a few other
                                            questions and issues that we
                                            have about the tests
                                            themselves.</div>
                                          <br>
                                        </div>
                                        <div>1. The test suite fails to
                                          build with a couple warnings.<br>
                                        </div>
                                        <div><br>
                                        </div>
                                        <div>Below is the stderr log
                                          from compilation:<br>
                                        </div>
                                        <br>
                                        <blockquote class="gmail_quote"
                                          style="margin:0px 0px 0px
                                          0.8ex;border-left:1px solid
                                          rgb(204,204,204);padding-left:1ex">FAILED:
                                          <a
                                            href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                            target="_blank"
                                            moz-do-not-send="true"
                                            class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
                                          cc
                                          -Ilib/76b5a35@@ts_dpdk_pmd@sta
                                          -Ilib -I../../lib
                                          -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
                                          -fdiagnostics-color=always
                                          -pipe -D_FILE_OFFSET_BITS=64
                                          -Wall -Winvalid-pch -Werror -g
                                          -D_GNU_SOURCE -O0 -ggdb -Wall
                                          -W -fPIC -MD -MQ '<a
                                            href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                            target="_blank"
                                            moz-do-not-send="true"
                                            class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
                                          -MF '<a
                                            href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d"
                                            target="_blank"
                                            moz-do-not-send="true"
                                            class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
                                          -o '<a
                                            href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                            target="_blank"
                                            moz-do-not-send="true"
                                            class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
                                          -c ../../lib/dpdk_pmd_ts.c<br>
                                          ../../lib/dpdk_pmd_ts.c: In
                                          function
                                          ‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
                                          no format arguments
                                          [-Werror=format-security]<br>
                                          5577 |     rc =
                                          te_kvpair_add(result, buf,
                                          mode);<br>
                                          |     ^~<br>
                                          cc1: all warnings being
                                          treated as errors<br>
                                          ninja: build stopped:
                                          subcommand failed.<br>
                                          ninja: Entering directory `.'<br>
                                          FAILED: <a
                                            href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                            target="_blank"
                                            moz-do-not-send="true"
                                            class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
                                          cc
                                          -Ilib/76b5a35@@ts_dpdk_pmd@sta
                                          -Ilib -I../../lib
                                          -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
                                          -fdiagnostics-color=always
                                          -pipe -D_FILE_OFFSET_BITS=64
                                          -Wall -Winvalid-pch -Werror -g
                                          -D_GNU_SOURCE -O0 -ggdb -Wall
                                          -W -fPIC -MD -MQ '<a
                                            href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                            target="_blank"
                                            moz-do-not-send="true"
                                            class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
                                          -MF '<a
                                            href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d"
                                            target="_blank"
                                            moz-do-not-send="true"
                                            class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
                                          -o '<a
                                            href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                            target="_blank"
                                            moz-do-not-send="true"
                                            class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
                                          -c ../../lib/dpdk_pmd_ts.c<br>
                                          ../../lib/dpdk_pmd_ts.c: In
                                          function
                                          ‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
                                          no format arguments
                                          [-Werror=format-security]<br>
                                          5577 |     rc =
                                          te_kvpair_add(result, buf,
                                          mode);<br>
                                          |     ^~<br>
                                          cc1: all warnings being
                                          treated as errors<br>
                                        </blockquote>
                                        <div>
                                          <div>
                                            <div><br>
                                            </div>
                                            <div>This error wasn't
                                              occurring last week, which
                                              was the last time I ran
                                              the tests.<br>
                                            </div>
                                            <div>The TE host and the DUT
                                              have GCC v9.4.0 installed,
                                              and the tester has GCC
                                              v11.4.0 installed, if this
                                              information is helpful.<br>
                                            </div>
                                            <div><br>
                                            </div>
                                            <div>2. On the Mellanox
                                              CX5s, there are over 6,000
                                              tests run, which
                                              collectively take around 9
                                              hours. Is it possible, and
                                              would it make sense, to
                                              lower the test coverage
                                              and have the test suite
                                              run faster?<br>
                                              <br>
                                            </div>
                                            <div>For some context, we
                                              run immediate testing on
                                              incoming patches for DPDK
                                              main and development
                                              branches, as well as
                                              periodic test runs on the
                                              main, stable, and LTS
                                              branches.<br>
                                            </div>
                                            <div>For us to consider
                                              including this test suite
                                              as part of our immediate
                                              testing on patches, we
                                              would have to reduce the
                                              test coverage to the most
                                              important tests.<br>
                                              This is primarily to
                                              reduce the testing time
                                              to, for example, less than
                                              30 minutes. Testing on
                                              patches can't take too
                                              long because the lab can
                                              receive numerous patches
                                              each day, which each
                                              require individual testing
                                              runs.<br>
                                              <br>
                                            </div>
                                            <div>At what frequency we
                                              run these tests, and on
                                              what, still needs to be
                                              discussed with the DPDK
                                              community, but it would be
                                              nice to know if the test
                                              suite had a "dial" to
                                              control the testing
                                              coverage.<br>
                                            </div>
                                            <div><br>
                                            </div>
                                            <div>3. We see a lot of test
                                              failures on our Mellanox
                                              CX5 NICs. Around 2,300 of
                                              ~6,600 tests passed. Is
                                              there anything we can do
                                              to diagnose these test
                                              failures?<br>
                                            </div>
                                            <div><br>
                                            </div>
                                            <div>Thanks,<br>
                                            </div>
                                            <div>Adam<br>
                                            </div>
                                            <div><br>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                      <br>
                                      <div class="gmail_quote">
                                        <div dir="ltr"
                                          class="gmail_attr">On Tue, Aug
                                          29, 2023 at 8:07 AM Andrew
                                          Rybchenko <<a
                                            href="mailto:andrew.rybchenko@oktetlabs.ru"
                                            target="_blank"
                                            moz-do-not-send="true"
                                            class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
                                          wrote:<br>
                                        </div>
                                        <blockquote class="gmail_quote"
                                          style="margin:0px 0px 0px
                                          0.8ex;border-left:1px solid
                                          rgb(204,204,204);padding-left:1ex">
                                          <div>
                                            <div>Hi Adam,<br>
                                              <br>
                                              I've pushed the fix in
                                              main branch and a new tag
                                              v1.18.1. It should solve
                                              the problem with IPv6
                                              address from DNS.<br>
                                              <br>
                                              Andrew.<br>
                                              <br>
                                              On 8/29/23 00:05, Andrew
                                              Rybchenko wrote:<br>
                                            </div>
                                            <blockquote type="cite">
                                              <div>Hi Adam,<br>
                                                <br>
                                                > Does the test
                                                engine prefer to use
                                                IPv6 over IPv4 for
                                                initiating the RCF
                                                connection to the test
                                                bed hosts? And if so, is
                                                there a way to force it
                                                to use IPv4?<br>
                                                <br>
                                                Brilliant idea. If DNS
                                                returns both IPv4 and
                                                IPv6 addresses in your
                                                case, I guess it is the
                                                root cause of the
                                                problem.<br>
                                                Of course, it is TE
                                                problem since I see
                                                really weird code in
                                                lib/comm_net_engine/comm_net_engine.c
                                                line 135.<br>
                                                <br>
                                                I've pushed fix to the
                                                branch
                                                user/arybchik/fix_ipv4_only
                                                in
                                                ts-factory/test-environment
                                                repository. Please, try.<br>
                                                <br>
                                                It is late night fix
                                                with minimal testing and
                                                no review. I'll pass it
                                                through review process
                                                tomorrow and<br>
                                                hopefully it will be
                                                released in one-two
                                                days.<br>
                                                <br>
                                                Andrew.<br>
                                                <br>
                                                On 8/28/23 18:02, Adam
                                                Hassick wrote:<br>
                                              </div>
                                              <blockquote type="cite">
                                                <div dir="ltr">
                                                  <div>
                                                    <div>
                                                      <div>Hi Andrew,<br>
                                                        <br>
                                                      </div>
                                                      We have yet to
                                                      notice a distinct
                                                      pattern with the
                                                      failures.
                                                      Sometimes, the RCF
                                                      will start and
                                                      connect without
                                                      issue a few times
                                                      in a row before
                                                      failing to connect
                                                      again. Once the
                                                      issue begins to
                                                      occur, neither
                                                      rebooting all of
                                                      the hosts (test
                                                      engine VM, tester,
                                                      IUT) or deleting
                                                      all of the build
                                                      directories
                                                      (suites, agents,
                                                      inst) and
                                                      rebooting the
                                                      hosts afterward
                                                      resolves the
                                                      issue. When it
                                                      begins working
                                                      again seems very
                                                      arbitrary to us.<br>
                                                      <br>
                                                    </div>
                                                    <div>I do usually
                                                      try to terminate
                                                      the test engine
                                                      with Ctrl+C, but
                                                      when it hangs
                                                      while trying to
                                                      start RCF, that
                                                      does not work.<br>
                                                    </div>
                                                    <div><br>
                                                    </div>
                                                    <div>Does the test
                                                      engine prefer to
                                                      use IPv6 over IPv4
                                                      for initiating the
                                                      RCF connection to
                                                      the test bed
                                                      hosts? And if so,
                                                      is there a way to
                                                      force it to use
                                                      IPv4?<br>
                                                      <br>
                                                    </div>
                                                    <div> - Adam<br>
                                                    </div>
                                                  </div>
                                                </div>
                                                <br>
                                                <div class="gmail_quote">
                                                  <div dir="ltr"
                                                    class="gmail_attr">On
                                                    Fri, Aug 25, 2023 at
                                                    1:35 PM Andrew
                                                    Rybchenko <<a
                                                      href="mailto:andrew.rybchenko@oktetlabs.ru"
                                                      target="_blank"
                                                      moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
                                                    wrote:<br>
                                                  </div>
                                                  <blockquote
                                                    class="gmail_quote"
                                                    style="margin:0px
                                                    0px 0px
                                                    0.8ex;border-left:1px
                                                    solid
                                                    rgb(204,204,204);padding-left:1ex">
                                                    <div>
                                                      <div>> I'll
                                                        double-check
                                                        test engine on
                                                        Ubuntu 20.04 and
                                                        Ubuntu 22.04.<br>
                                                        <br>
                                                        Done. It works
                                                        fine for me
                                                        without any
                                                        issues.<br>
                                                        <br>
                                                        Have you noticed
                                                        any pattern when
                                                        it works or does
                                                        not work?<br>
                                                        May be it is a
                                                        problem of not
                                                        clean state
                                                        after
                                                        termination?<br>
                                                        Does it work
                                                        fine the first
                                                        time after DUTs
                                                        reboot?<br>
                                                        How do you
                                                        terminate
                                                        testing? It
                                                        should be done
                                                        using Ctrl+C in
                                                        terminal where
                                                        you execute
                                                        run.sh command.<br>
                                                         In this case it
                                                        should shutdown
                                                        gracefully and
                                                        close all test
                                                        agents and
                                                        engine
                                                        applications.<br>
                                                        <br>
                                                        (I'm trying to
                                                        understand why
                                                        you've seen many
                                                        test agent
                                                        processes. It
                                                        should not
                                                        happen.)<br>
                                                        <br>
                                                        Andrew.<br>
                                                        <br>
                                                        On 8/25/23
                                                        17:41, Andrew
                                                        Rybchenko wrote:<br>
                                                      </div>
                                                      <blockquote
                                                        type="cite">
                                                        <div>On 8/25/23
                                                          17:06, Adam
                                                          Hassick wrote:<br>
                                                        </div>
                                                        <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>Hi
                                                          Andrew,<br>
                                                          <br>
                                                          </div>
                                                          Two of our
                                                          systems (the
                                                          Test Engine
                                                          runner and the
                                                          DUT host) are
                                                          running Ubuntu
                                                          20.04 LTS,
                                                          however this
                                                          morning I
                                                          noticed that
                                                          the tester
                                                          system (the
                                                          one having
                                                          issues) is
                                                          running Ubuntu
                                                          22.04 LTS.<br>
                                                          </div>
                                                          <div>This
                                                          could be the
                                                          source of the
                                                          problem. I
                                                          encountered a
                                                          dependency
                                                          issue trying
                                                          to run the
                                                          Test Engine on
                                                          22.04 LTS, so
                                                          I downgraded
                                                          the system.
                                                          Since the
                                                          tester is also
                                                          the host
                                                          having
                                                          connection
                                                          issues, I will
                                                          try
                                                          downgrading
                                                          that system to
                                                          20.04, and see
                                                          if that
                                                          changes
                                                          anything.<br>
                                                          </div>
                                                          </div>
                                                        </blockquote>
                                                        <br>
                                                        Unlikely, but
                                                        who knows. We
                                                        run tests (DUTs)
                                                        on Ubuntu 20.04,
                                                        Ubuntu 22.04,
                                                        Ubuntu 22.10,
                                                        Ubuntu 23.04,
                                                        Debian 11 and
                                                        Fedora 38 every
                                                        night.<br>
                                                        Right now Debian
                                                        11 is used for
                                                        test engine in
                                                        nightly
                                                        regressions.<br>
                                                        <br>
                                                        I'll
                                                        double-check
                                                        test engine on
                                                        Ubuntu 20.04 and
                                                        Ubuntu 22.04.<br>
                                                        <br>
                                                        <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div>I did try
                                                          passing in the
                                                          "--vg-rcf"
                                                          argument to
                                                          the run.sh
                                                          script of the
                                                          test suite
                                                          after
                                                          installing
                                                          valgrind, but
                                                          there was no
                                                          additional
                                                          output that I
                                                          saw.<br>
                                                          </div>
                                                          </div>
                                                        </blockquote>
                                                        <br>
                                                        Sorry, I should
                                                        valgrind output
                                                        should be in
                                                        valgrind.te_rcf
                                                        (direction where
                                                        you run test
                                                        engine).<br>
                                                        <br>
                                                        <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div><br>
                                                          </div>
                                                          <div>I will
                                                          try pulling in
                                                          the changes
                                                          you've pushed
                                                          up, and will
                                                          see if that
                                                          fixes
                                                          anything.<br>
                                                          <br>
                                                          </div>
                                                          <div>Thanks,<br>
                                                          </div>
                                                          <div>Adam<br>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">
                                                          <div dir="ltr"
class="gmail_attr">On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div>
                                                          <div>Hello
                                                          Adam, <br>
                                                          <br>
                                                          On 8/24/23
                                                          23:54, Andrew
                                                          Rybchenko
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">I'd
                                                          like to try to
                                                          repeat the
                                                          problem
                                                          locally. Which
                                                          Linux distro
                                                          is running on
                                                          test engine
                                                          and agents? <br>
                                                          <br>
                                                          In fact I know
                                                          one problem
                                                          with Debian 12
                                                          and Fedora 38
                                                          and we have <br>
                                                          patch in
                                                          review to fix
                                                          it, however,
                                                          the behaviour
                                                          is different
                                                          in <br>
                                                          this case, so
                                                          it is unlike
                                                          the same
                                                          problem. <br>
                                                          </blockquote>
                                                          <br>
                                                          I've just
                                                          published a
                                                          new tag which
                                                          fixes known
                                                          test engine
                                                          side problems
                                                          on Debian 12
                                                          and Fedora 38.<br>
                                                          <br>
                                                          <blockquote
                                                          type="cite"> <br>
                                                          One more idea
                                                          is to install
                                                          valgrind on
                                                          the test
                                                          engine host
                                                          and <br>
                                                          run with
                                                          option
                                                          --vg-rcf to
                                                          check if
                                                          something
                                                          weird is
                                                          happening. <br>
                                                          <br>
                                                          What I don't
                                                          understand
                                                          right now is
                                                          why I see just
                                                          one failed
                                                          attempt <br>
                                                          to connect in
                                                          your log.txt
                                                          and then
                                                          Logger
                                                          shutdown after
                                                          9 minutes. <br>
                                                          <br>
                                                          Andrew. <br>
                                                          <br>
                                                          On 8/24/23
                                                          23:29, Adam
                                                          Hassick wrote:
                                                          <br>
                                                          <blockquote
                                                          type="cite"> >
                                                          Is there any
                                                          firewall in
                                                          the network or
                                                          on test hosts
                                                          which could
                                                          block incoming
                                                          TCP connection
                                                          to the port
                                                          23571 <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          from the host
                                                          where you run
                                                          test engine? <br>
                                                          <br>
                                                          Our test
                                                          engine host
                                                          and the
                                                          testbed are on
                                                          the same
                                                          subnet. The
                                                          connection
                                                          does work
                                                          sometimes. <br>
                                                          <br>
                                                           > If
                                                          behaviour the
                                                          same on the
                                                          next try and
                                                          you see that
                                                          test agent is
                                                          kept running,
                                                          could you
                                                          check using <br>
                                                           > <br>
                                                           > #
                                                          netstat -tnlp
                                                          <br>
                                                           > <br>
                                                           > that
                                                          Test Agent is
                                                          listening on
                                                          the port and
                                                          try to
                                                          establish TCP
                                                          connection
                                                          from test
                                                          agent using <br>
                                                           > <br>
                                                           > $ telnet
                                                          <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu"
target="_blank" moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a>
                                                          <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          23571 <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          <br>
                                                           > <br>
                                                           > and
                                                          check if TCP
                                                          connection
                                                          could be
                                                          established. <br>
                                                          <br>
                                                          I was able to
                                                          replicate the
                                                          same behavior
                                                          again, where
                                                          it hangs while
                                                          RCF is trying
                                                          to start. <br>
                                                          Running this
                                                          command, I see
                                                          this in the
                                                          output: <br>
                                                          <br>
                                                          tcp        0  
                                                             0 <a
                                                          href="http://0.0.0.0:23571"
target="_blank" moz-do-not-send="true">0.0.0.0:23571</a> <a
                                                          href="http://0.0.0.0:23571"
target="_blank" moz-do-not-send="true"><http://0.0.0.0:23571></a>          
                                                          0.0.0.0:*    
                                                                   
                                                          LISTEN    
                                                           18599/ta <br>
                                                          <br>
                                                          So it seems
                                                          like it is
                                                          listening on
                                                          the correct
                                                          port. <br>
                                                          Additionally,
                                                          I was able to
                                                          connect to the
                                                          Tester machine
                                                          from our Test
                                                          Engine host
                                                          using telnet.
                                                          It printed the
                                                          PID of the
                                                          process once
                                                          the connection
                                                          was opened. <br>
                                                          <br>
                                                          I tried
                                                          running the
                                                          "ta"
                                                          application
                                                          manually on
                                                          the command
                                                          line, and it
                                                          didn't print
                                                          anything at
                                                          all. <br>
                                                          Maybe the
                                                          issue is
                                                          something on
                                                          the Test
                                                          Engine side. <br>
                                                          <br>
                                                          On Thu, Aug
                                                          24, 2023 at
                                                          2:35 PM Andrew
                                                          Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a> <a
                                                          href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
                                                          wrote: <br>
                                                          <br>
                                                              Hi Adam, <br>
                                                          <br>
                                                               > On
                                                          the tester
                                                          host (which
                                                          appears to be
                                                          the Peer
                                                          agent), there
                                                          <br>
                                                              are four
                                                          processes that
                                                          I see running,
                                                          which look
                                                          like the test
                                                          <br>
                                                              agent
                                                          processes. <br>
                                                          <br>
                                                              Before the
                                                          next try I'd
                                                          recommend to
                                                          kill these
                                                          processes. <br>
                                                          <br>
                                                              Is there
                                                          any firewall
                                                          in the network
                                                          or on test
                                                          hosts which
                                                          could <br>
                                                              block
                                                          incoming TCP
                                                          connection to
                                                          the port 23571
                                                          <br>
                                                              <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          from the host
                                                          <br>
                                                              where you
                                                          run test
                                                          engine? <br>
                                                          <br>
                                                              If
                                                          behaviour the
                                                          same on the
                                                          next try and
                                                          you see that
                                                          test agent is
                                                          <br>
                                                              kept
                                                          running, could
                                                          you check
                                                          using <br>
                                                          <br>
                                                              # netstat
                                                          -tnlp <br>
                                                          <br>
                                                              that Test
                                                          Agent is
                                                          listening on
                                                          the port and
                                                          try to
                                                          establish TCP
                                                          <br>
                                                              connection
                                                          from test
                                                          agent using <br>
                                                          <br>
                                                              $ telnet <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a>
                                                          <br>
                                                              <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          23571 <br>
                                                              <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          <br>
                                                          <br>
                                                              and check
                                                          if TCP
                                                          connection
                                                          could be
                                                          established. <br>
                                                          <br>
                                                              Another
                                                          idea is to
                                                          login Tester
                                                          under root as
                                                          testing does,
                                                          get <br>
                                                              start TA
                                                          command from
                                                          the log and
                                                          try it by
                                                          hands without
                                                          -n and <br>
                                                              remove
                                                          extra
                                                          escaping. <br>
                                                          <br>
                                                              # sudo
                                                          PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
                                                          <br>
                                                             
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=<br>
                                                          <br>
                                                              Hopefully
                                                          in this case
                                                          test agent
                                                          directory
                                                          remains in the
                                                          /tmp and <br>
                                                              you don't
                                                          need to copy
                                                          it as testing
                                                          does. <br>
                                                              May be
                                                          output could
                                                          shed some
                                                          light on
                                                          what's going
                                                          on. <br>
                                                          <br>
                                                              Andrew. <br>
                                                          <br>
                                                              On 8/24/23
                                                          17:30, Adam
                                                          Hassick wrote:
                                                          <br>
                                                          <blockquote
                                                          type="cite">   
                                                          Hi Andrew, <br>
                                                          <br>
                                                              This is
                                                          the output
                                                          that I see in
                                                          the terminal
                                                          when this
                                                          failure <br>
                                                              occurs,
                                                          after the test
                                                          agent binaries
                                                          build and the
                                                          test engine <br>
                                                              starts: <br>
                                                          <br>
                                                              Platform
                                                          default build
                                                          - pass <br>
                                                              Simple RCF
                                                          consistency
                                                          check
                                                          succeeded <br>
                                                             
                                                          --->>>
                                                          Starting
                                                          Logger...done
                                                          <br>
                                                             
                                                          --->>>
                                                          Starting
                                                          RCF...rcf_net_engine_connect():
                                                          Connection
                                                          timed <br>
                                                              out <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu:23571</a>
                                                          <br>
                                                              <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          <br>
                                                          <br>
                                                              Then, it
                                                          hangs here
                                                          until I kill
                                                          the "te_rcf"
                                                          and "te_tee" <br>
                                                              processes.
                                                          I let it hang
                                                          for around 9
                                                          minutes. <br>
                                                          <br>
                                                              On the
                                                          tester host
                                                          (which appears
                                                          to be the Peer
                                                          agent), there
                                                          are <br>
                                                              four
                                                          processes that
                                                          I see running,
                                                          which look
                                                          like the test
                                                          agent <br>
                                                              processes.
                                                          <br>
                                                          <br>
                                                              ta.Peer is
                                                          an empty file.
                                                          I've attached
                                                          the log.txt
                                                          from this run.
                                                          <br>
                                                          <br>
                                                               - Adam <br>
                                                          <br>
                                                              On Thu,
                                                          Aug 24, 2023
                                                          at 4:22 AM
                                                          Andrew
                                                          Rybchenko <br>
                                                              <<a
                                                          href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
                                                          <br>
                                                              <a
                                                          href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
                                                          wrote: <br>
                                                          <br>
                                                                  Hi
                                                          Adam, <br>
                                                          <br>
                                                                  Yes,
                                                          TE_RCFUNIX_TIMEOUT
                                                          is in seconds.
                                                          I've
                                                          double-checked
                                                          <br>
                                                                  that
                                                          it goes to
                                                          'copy_timeout'
                                                          in
                                                          ts-conf/rcf.conf.
                                                          <br>
                                                                 
                                                          Description in
                                                          in
                                                          doc/sphinx/pages/group_te_engine_rcf.rst
                                                          <br>
                                                                  says
                                                          that
                                                          copy_timeout
                                                          is in seconds
                                                          and
                                                          implementation
                                                          in <br>
                                                                 
                                                          lib/rcfunix/rcfunix.c
                                                          passes the
                                                          value to
                                                          select()
                                                          tv_sec. <br>
                                                                 
                                                          Theoretically
                                                          select() could
                                                          be interrupted
                                                          by signal, but
                                                          I <br>
                                                                  think
                                                          it is unlikely
                                                          here. <br>
                                                          <br>
                                                                  I'm
                                                          not sure that
                                                          I understand
                                                          what do you
                                                          mean by RCF <br>
                                                                 
                                                          connection
                                                          timeout. Does
                                                          it happen on
                                                          TE startup
                                                          when RCF <br>
                                                                  starts
                                                          test agents.
                                                          If so,
                                                          TE_RCFUNIX_TIMEOUT
                                                          could help. Or
                                                          <br>
                                                                  does
                                                          it happen when
                                                          tests are in
                                                          progress, e.g.
                                                          in the middle
                                                          <br>
                                                                  of a
                                                          test. If so,
                                                          TE_RCFUNIX_TIMEOUT
                                                          is unrelated
                                                          and most <br>
                                                                  likely
                                                          either host
                                                          with test
                                                          agent dies or
                                                          test agent
                                                          itself <br>
                                                                 
                                                          crashes. It
                                                          would be
                                                          easier for me
                                                          if classify it
                                                          if you share <br>
                                                                  text
                                                          log (log.txt,
                                                          full or just
                                                          corresponding
                                                          fragment with
                                                          <br>
                                                                  some
                                                          context). Also
                                                          content of
                                                          ta.DPDK or
                                                          ta.Peer file <br>
                                                                 
                                                          depending on
                                                          which agent
                                                          has problems
                                                          could shed
                                                          some light. <br>
                                                                 
                                                          Corresponding
                                                          files contain
                                                          stdout/stderr
                                                          of test
                                                          agents. <br>
                                                          <br>
                                                                 
                                                          Andrew. <br>
                                                          <br>
                                                                  On
                                                          8/23/23 17:45,
                                                          Adam Hassick
                                                          wrote: <br>
                                                          <blockquote
                                                          type="cite">       
                                                          Hi Andrew, <br>
                                                          <br>
                                                                  I've
                                                          set up a test
                                                          rig repository
                                                          here, and have
                                                          created <br>
                                                                 
                                                          configurations
                                                          for our
                                                          development
                                                          testbed based
                                                          off of the <br>
                                                                 
                                                          examples. <br>
                                                                  We've
                                                          been able to
                                                          get the test
                                                          suite to run
                                                          manually on <br>
                                                                 
                                                          Mellanox CX5
                                                          devices once.
                                                          <br>
                                                                 
                                                          However, we
                                                          are running
                                                          into an issue
                                                          where, when
                                                          RCF starts, <br>
                                                                  the
                                                          RCF connection
                                                          times out very
                                                          frequently. We
                                                          aren't sure <br>
                                                                  why
                                                          this is the
                                                          case. <br>
                                                                  It
                                                          works
                                                          sometimes, but
                                                          most of the
                                                          time when we
                                                          try to run <br>
                                                                  the
                                                          test engine,
                                                          it encounters
                                                          this issue. <br>
                                                                  I've
                                                          tried changing
                                                          the RCF port
                                                          by setting <br>
                                                                 
                                                          "TE_RCF_PORT=<some
                                                          port
                                                          number>"
                                                          and rebooting
                                                          the testbed <br>
                                                                 
                                                          machines.
                                                          Neither seems
                                                          to fix the
                                                          issue. <br>
                                                          <br>
                                                                  It
                                                          also seems
                                                          like the
                                                          timeout takes
                                                          far longer
                                                          than 60 <br>
                                                                 
                                                          seconds, even
                                                          when running
                                                          "export
                                                          TE_RCFUNIX_TIMEOUT=60"
                                                          <br>
                                                                  before
                                                          I try to run
                                                          the test
                                                          suite. <br>
                                                                  I
                                                          assume the
                                                          unit for this
                                                          variable is
                                                          seconds? <br>
                                                          <br>
                                                                 
                                                          Thanks, <br>
                                                                  Adam <br>
                                                          <br>
                                                                  On
                                                          Mon, Aug 21,
                                                          2023 at
                                                          10:19 AM Adam
                                                          Hassick <br>
                                                                  <<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a> <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>>
                                                          wrote: <br>
                                                          <br>
                                                                      Hi
                                                          Andrew, <br>
                                                          <br>
                                                                     
                                                          Thanks, I've
                                                          cloned the
                                                          example
                                                          repository and
                                                          will start <br>
                                                                     
                                                          setting up a
                                                          configuration
                                                          for our
                                                          development
                                                          testbed <br>
                                                                     
                                                          today. I'll
                                                          let you know
                                                          if I run into
                                                          any
                                                          difficulties <br>
                                                                      or
                                                          have any
                                                          questions. <br>
                                                          <br>
                                                                       -
                                                          Adam <br>
                                                          <br>
                                                                      On
                                                          Sun, Aug 20,
                                                          2023 at
                                                          4:40 AM Andrew
                                                          Rybchenko <br>
                                                                     
                                                          <<a
                                                          href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
                                                          <br>
                                                                      <a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
                                                          moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
                                                          wrote: <br>
                                                          <br>
                Hi Adam, <br>
                                                          <br>
                I've published <br>
                <a href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ts-factory/ts-rigs-sample</a>
                                                          <br>
                <a href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/ts-rigs-sample></a>.
                                                          <br>
                Hopefully it will help to define your test rigs and <br>
                successfully run some tests manually. Feel free to <br>
                ask any questions and I'll answer here and try to <br>
                update documentation. <br>
                                                          <br>
                Meanwhile I'll prepare missing bits for steps (2) and <br>
                (3). <br>
                Hopefully everything is in place for step (4), but we <br>
                need to make steps (2) and (3) first. <br>
                                                          <br>
                Andrew. <br>
                                                          <br>
                On 8/18/23 21:40, Andrew Rybchenko wrote: <br>
                                                          <blockquote
                                                          type="cite">               
                                                          Hi Adam, <br>
                                                          <br>
                > I've conferred with the rest of the team, and we <br>
                think it would be best to move forward with mainly <br>
                option B. <br>
                                                          <br>
                OK, I'll provide the sample on Monday for you. It is <br>
                almost ready right now, but I need to double-check <br>
                it before publishing. <br>
                                                          <br>
                Regards, <br>
                Andrew. <br>
                                                          <br>
                On 8/17/23 20:03, Adam Hassick wrote: <br>
                                                          <blockquote
                                                          type="cite">               
                                                          Hi Andrew, <br>
                                                          <br>
                I'm adding the CI mailing list to this <br>
                conversation. Others in the community might find <br>
                this conversation valuable. <br>
                                                          <br>
                We do want to run testing on a regular basis. The <br>
                Jenkins integration will be very useful for us, as <br>
                most of our CI is orchestrated by Jenkins. <br>
                I've conferred with the rest of the team, and we <br>
                think it would be best to move forward with mainly <br>
                option B. <br>
                If you would like to know anything about our <br>
                testbeds that would help you with creating an <br>
                example ts-rigs repo, I'd be happy to answer any <br>
                questions you have. <br>
                                                          <br>
                We have multiple test rigs (we call these <br>
                "DUT-tester pairs") that we run our existing <br>
                hardware testing on, with differing network <br>
                hardware and CPU architecture. I figured this might <br>
                be an important detail. <br>
                                                          <br>
                Thanks, <br>
                Adam <br>
                                                          <br>
                On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <br>
                <<a href="mailto:andrew.rybchenko@oktetlabs.ru"
                                                          target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
                                                          <br>
                <a href="mailto:andrew.rybchenko@oktetlabs.ru"
                                                          target="_blank"
moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
                                                          wrote: <br>
                                                          <br>
                    Greatings Adam, <br>
                                                          <br>
                    I'm happy to hear that you're trying to bring <br>
                    it up. <br>
                                                          <br>
                    As I understand the final goal is to run it on <br>
                    regular basis. So, we need to make it properly <br>
                    from the very beginning. <br>
                    Bring up of all features consists of 4 steps: <br>
                                                          <br>
                    1. Create site-specific repository (we call it <br>
                    ts-rigs) which contains information about test <br>
                    rigs and other site-specific information like <br>
                    where to send mails, where to store logs etc. <br>
                    It is required for manual execution as well, <br>
                    since test rigs description is essential. I'll <br>
                    return to the topic below. <br>
                                                          <br>
                    2. Setup logs storage for automated runs. <br>
                    Basically it is a disk space plus apache2 web <br>
                    server with few CGI scripts which help a lot to <br>
                    save disk space. <br>
                                                          <br>
                    3. Setup Bublik web application which provides <br>
                    web interface to view testing results. Same as <br>
                    <a href="https://ts-factory.io/bublik"
                                                          target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://ts-factory.io/bublik</a>
                                                          <br>
                    <a href="https://ts-factory.io/bublik"
                                                          target="_blank"
moz-do-not-send="true"><https://ts-factory.io/bublik></a> <br>
                                                          <br>
                    4. Setup Jenkins to run tests on regularly, <br>
                    save logs in log storage (2) and import it to <br>
                    bublik (3). <br>
                                                          <br>
                    Last few month we spent on our homework to make <br>
                    it simpler to bring up automated execution <br>
                    using Jenkins - <br>
                    <a href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ts-factory/te-jenkins</a>
                                                          <br>
                    <a href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/te-jenkins></a>
                                                          <br>
                    Corresponding bits in dpdk-ethdev-ts will be <br>
                    available tomorrow. <br>
                                                          <br>
                    Let's return to the step (1). <br>
                                                          <br>
                    Unfortunately there is no publicly available <br>
                    example of the ts-rigs repository since <br>
                    sensitive site-specific information is located <br>
                    there. But I'm ready to help you to create it <br>
                    for UNH. I see two options here: <br>
                                                          <br>
                    (A) I'll ask questions and based on your <br>
                    answers will create the first draft with my <br>
                    comments. <br>
                                                          <br>
                    (B) I'll make a template/example ts-rigs repo, <br>
                    publish it and you'll create UNH ts-rigs based <br>
                    on it. <br>
                                                          <br>
                    Of course, I'll help to debug and finally bring <br>
                    it up in any case. <br>
                                                          <br>
                    (A) is a bit simpler for me and you, but (B) is <br>
                    a bit more generic and will help other <br>
                    potential users to bring it up. <br>
                    We can combine (A)+(B). I.e. start from (A). <br>
                    What do you think? <br>
                                                          <br>
                    Thanks, <br>
                    Andrew. <br>
                                                          <br>
                    On 8/17/23 15:18, Konstantin Ushakov wrote: <br>
                                                          <blockquote
                                                          type="cite">                   
                                                          Greetings
                                                          Adam, <br>
                                                          <br>
                                                          <br>
                    Thanks for contacting us. I copy Andrew who <br>
                    would be happy to help <br>
                                                          <br>
                    Thanks, <br>
                    Konstantin <br>
                                                          <br>
                                                          <blockquote
                                                          type="cite">                   
                                                          On 16 Aug
                                                          2023, at
                                                          21:50, Adam
                                                          Hassick <br>
                    <a href="mailto:ahassick@iol.unh.edu"
                                                          target="_blank"
moz-do-not-send="true"><ahassick@iol.unh.edu></a> <br>
                    <a href="mailto:ahassick@iol.unh.edu"
                                                          target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a> wrote: <br>
                                                          <br>
                     <br>
                    Greetings Konstantin, <br>
                                                          <br>
                    I am in the process of setting up the DPDK <br>
                    Poll Mode Driver test suite as an addition to <br>
                    our testing coverage for DPDK at the UNH lab. <br>
                                                          <br>
                    I have some questions about how to set the <br>
                    test suite arguments. <br>
                                                          <br>
                    I have been able to configure the Test Engine <br>
                    to connect to the hosts in the testbed. The <br>
                    RCF, Configurator, and Tester all begin to <br>
                    run, however the prelude of the test suite <br>
                    fails to run. <br>
                                                          <br>
                    <a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters</a>
                                                          <a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true"><https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters></a>
                                                          <br>
                                                          <br>
                    The documentation mentions that there are <br>
                    several test parameters for the test suite, <br>
                    like for the IUT test link MAC, etc. These <br>
                    seem like they would need to be set somewhere <br>
                    to run many of the tests. <br>
                                                          <br>
                    I see in the Test Engine documentation, there <br>
                    are instructions on how to create new <br>
                    parameters for test suites in the Tester <br>
                    configuration, but there is nothing in the <br>
                    user guide or in the Tester guide for how to <br>
                    set the arguments for the parameters when <br>
                    running the test suite that I can find. I'm <br>
                    not sure if I need to write my own Tester <br>
                    config, or if I should be setting these in <br>
                    some other way. <br>
                                                          <br>
                    How should these values be set? <br>
                                                          <br>
                    I'm also not sure what environment <br>
                    variables/arguments are strictly necessary or <br>
                    which are optional. <br>
                                                          <br>
                    Regards, <br>
                    Adam <br>
                                                          <br>
                    --                     *Adam Hassick* <br>
                    Senior Developer <br>
                    UNH InterOperability Lab <br>
                    <a href="mailto:ahassick@iol.unh.edu"
                                                          target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
                                                          <br>
                    <a href="mailto:ahassick@iol.unh.edu"
                                                          target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a> <br>
                    <a href="http://iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true">iol.unh.edu</a>
                                                          <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                    +1 (603) 475-8248 <br>
                                                          </blockquote>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <br>
                --                 *Adam Hassick* <br>
                Senior Developer <br>
                UNH InterOperability Lab <br>
                <a href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a> <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                <a href="http://iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true">iol.unh.edu</a>
                                                          <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                +1 (603) 475-8248 <br>
                                                          </blockquote>
                                                          <br>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <br>
                                                                      --
                                                                     
                                                          *Adam Hassick*
                                                          <br>
                                                                     
                                                          Senior
                                                          Developer <br>
                                                                     
                                                          UNH
                                                          InterOperability
                                                          Lab <br>
                                                                      <a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a> <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                                                                      <a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
                                                          <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                                                                      +1
                                                          (603) 475-8248
                                                          <br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                                  --
                                                                  *Adam
                                                          Hassick* <br>
                                                                  Senior
                                                          Developer <br>
                                                                  UNH
                                                          InterOperability
                                                          Lab <br>
                                                                  <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
                                                          <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                                                                  <a
                                                          href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a> <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                                                                  +1
                                                          (603) 475-8248
                                                          <br>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <br>
                                                              --    
                                                          *Adam Hassick*
                                                          <br>
                                                              Senior
                                                          Developer <br>
                                                              UNH
                                                          InterOperability
                                                          Lab <br>
                                                              <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
                                                          <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                                                              <a
                                                          href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a> <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                                                              +1 (603)
                                                          475-8248 <br>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          -- <br>
                                                          *Adam Hassick*
                                                          <br>
                                                          Senior
                                                          Developer <br>
                                                          UNH
                                                          InterOperability
                                                          Lab <br>
                                                          <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
                                                          <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                                                          <a
                                                          href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a> <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                                                          +1 (603)
                                                          475-8248 <br>
                                                          </blockquote>
                                                          <br>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br
                                                          clear="all">
                                                          <br>
                                                          <span
                                                          class="gmail_signature_prefix">--
                                                          </span><br>
                                                          <div dir="ltr"
class="gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div><b><span
style="background-color:rgb(255,255,255)"><span
                                                          style="color:rgb(102,102,102)">Adam
                                                          Hassick</span></span></b><br>
                                                          </div>
                                                          <span
                                                          style="color:rgb(102,102,102)"></span></div>
                                                          <div><span
                                                          style="color:rgb(102,102,102)">Senior
                                                          Developer</span></div>
                                                          <div><span
                                                          style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
                                                          style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
                                                          style="color:rgb(102,102,102)"></span></div>
                                                          <div><span
                                                          style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                                                          </span></div>
                                                          <div><span
                                                          style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
                                                          </span></div>
                                                          +1 (603)
                                                          475-8248<br>
                                                          </div>
                                                          </div>
                                                        </blockquote>
                                                        <br>
                                                      </blockquote>
                                                      <br>
                                                    </div>
                                                  </blockquote>
                                                </div>
                                                <br clear="all">
                                                <br>
                                                <span
                                                  class="gmail_signature_prefix">--
                                                </span><br>
                                                <div dir="ltr"
                                                  class="gmail_signature">
                                                  <div dir="ltr">
                                                    <div>
                                                      <div><b><span
                                                          style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
                                                      </div>
                                                      <span
                                                        style="color:rgb(102,102,102)"></span></div>
                                                    <div><span
                                                        style="color:rgb(102,102,102)">Senior
                                                        Developer</span></div>
                                                    <div><span
                                                        style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
                                                          style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
                                                        style="color:rgb(102,102,102)"></span></div>
                                                    <div><span
                                                        style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                                                      </span></div>
                                                    <div><span
                                                        style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
                                                      </span></div>
                                                    +1 (603) 475-8248<br>
                                                  </div>
                                                </div>
                                              </blockquote>
                                              <br>
                                            </blockquote>
                                            <br>
                                          </div>
                                        </blockquote>
                                      </div>
                                      <br clear="all">
                                      <br>
                                      <span
                                        class="gmail_signature_prefix">--
                                      </span><br>
                                      <div dir="ltr"
                                        class="gmail_signature">
                                        <div dir="ltr">
                                          <div>
                                            <div><b><span
                                                  style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
                                            </div>
                                            <span
                                              style="color:rgb(102,102,102)"></span></div>
                                          <div><span
                                              style="color:rgb(102,102,102)">Senior
                                              Developer</span></div>
                                          <div><span
                                              style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
                                                  style="background-color:rgb(255,255,255)">UNH
                                                  InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
                                          <div><span
                                              style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                moz-do-not-send="true"
                                                class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                                            </span></div>
                                          <div><span
                                              style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
                                            </span></div>
                                          +1 (603) 475-8248<br>
                                        </div>
                                      </div>
                                    </blockquote>
                                    <br>
                                  </div>
                                </blockquote>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                        <br>
                      </div>
                    </blockquote>
                  </div>
                  <br clear="all">
                  <br>
                  <span class="gmail_signature_prefix">-- </span><br>
                  <div dir="ltr" class="gmail_signature">
                    <div dir="ltr">
                      <div>
                        <div><b><span
                              style="background-color:rgb(255,255,255)"><span
                                style="color:rgb(102,102,102)">Adam
                                Hassick</span></span></b><br>
                        </div>
                        <span style="color:rgb(102,102,102)"></span></div>
                      <div><span style="color:rgb(102,102,102)">Senior
                          Developer</span></div>
                      <div><span style="color:rgb(102,102,102)"><span
                            style="color:rgb(11,83,148)"><span
                              style="background-color:rgb(255,255,255)">UNH
                              InterOperability Lab</span></span></span><span
                          style="color:rgb(102,102,102)"></span></div>
                      <div><span style="color:rgb(102,102,102)"><a
                            href="mailto:ahassick@iol.unh.edu"
                            target="_blank" moz-do-not-send="true"
                            class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                        </span></div>
                      <div><span style="color:rgb(102,102,102)"><a
                            href="https://www.iol.unh.edu/"
                            target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
                        </span></div>
                      +1 (603) 475-8248<br>
                    </div>
                  </div>
                </blockquote>
                <br>
              </div>
            </blockquote>
          </div>
          <br clear="all">
          <br>
          <span class="gmail_signature_prefix">-- </span><br>
          <div dir="ltr" class="gmail_signature">
            <div dir="ltr">
              <div>
                <div><b><span style="background-color:rgb(255,255,255)"><span
                        style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
                </div>
                <span style="color:rgb(102,102,102)"></span></div>
              <div><span style="color:rgb(102,102,102)">Senior Developer</span></div>
              <div><span style="color:rgb(102,102,102)"><span
                    style="color:rgb(11,83,148)"><span
                      style="background-color:rgb(255,255,255)">UNH
                      InterOperability Lab</span></span></span><span
                  style="color:rgb(102,102,102)"></span></div>
              <div><span style="color:rgb(102,102,102)"><a
                    href="mailto:ahassick@iol.unh.edu" target="_blank"
                    moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                </span></div>
              <div><span style="color:rgb(102,102,102)"><a
                    href="https://www.iol.unh.edu/" target="_blank"
                    moz-do-not-send="true">iol.unh.edu</a><br>
                </span></div>
              +1 (603) 475-8248<br>
            </div>
          </div>
        </blockquote>
      </div>
      <br clear="all">
      <br>
      <span class="gmail_signature_prefix">-- </span><br>
      <div dir="ltr" class="gmail_signature">
        <div dir="ltr">
          <div>
            <div><b><span style="background-color:rgb(255,255,255)"><span
                    style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
            </div>
            <span style="color:rgb(102,102,102)"></span></div>
          <div><span style="color:rgb(102,102,102)">Senior Developer</span></div>
          <div><span style="color:rgb(102,102,102)"><span
                style="color:rgb(11,83,148)"><span
                  style="background-color:rgb(255,255,255)">UNH
                  InterOperability Lab</span></span></span><span
              style="color:rgb(102,102,102)"></span></div>
          <div><span style="color:rgb(102,102,102)"><a
                href="mailto:ahassick@iol.unh.edu" target="_blank"
                moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
            </span></div>
          <div><span style="color:rgb(102,102,102)"><a
                href="https://www.iol.unh.edu/" target="_blank"
                moz-do-not-send="true">iol.unh.edu</a><br>
            </span></div>
          +1 (603) 475-8248<br>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>