<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">Hi Adam,<br>
      <br>
      I've uploaded fresh testing results to ts-factory.io [1] to be on
      the same page.<br>
      <br>
      I think I know why your and mine results on Intel 710 series NICs
      differ so much. Testing results expectations database
      (dpdk-ethdev-ts/trc/*) is filled in in terms of TRC tags.  I.e.
      expectations depends on TRC tags discovered by helper scripts when
      testing is started. These tags identify various aspects of what is
      tested. Ideally expectations should be written in terms of root
      cause of the expected behaviour. If it is a driver expectations,
      driver tag should be used. If it is HW limitation, tags with PCI
      IDs should be used. However, it is not always easy to classify it
      correctly if you're not involved in driver development. So, in
      order case expectations for 710's Intel are filled in in terms of
      PCI IDs. I guess PCI ID differ in your case and that's why
      expectations filled in for my NIC do not apply to your runs.<br>
      <br>
      Just try to add the following option when you run on your 710's
      Intel in order to mimic mine and see if it helps to achieve better
      pass rate.<br>
      --trc-tag=pci-8086-1572<br>
      <br>
      BTW, fresh TE tag <span class="text-[0.625rem] font-medium
        leading-[1.125rem]">v1.21.0 has improved algorithm to choose
        tests for --tester-dial option. It should have better coverage
        now.</span><br>
      <br>
      Andrew.<br>
      <br>
      [1]
<a class="moz-txt-link-freetext" href="https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1">https://ts-factory.io/bublik/v2/runs?startDate=2023-09-16&finishDate=2023-09-16&runData=&runDataExpr=&page=1</a><br>
      <br>
      On 9/13/23 18:45, Andrew Rybchenko wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:ad89983f-70e4-2158-2104-7601d5c1528f@oktetlabs.ru">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <div class="moz-cite-prefix">Hi Adam,<br>
        <br>
        I've pushed new TE tag v1.20.0 which supported a new
        command-line option --tester-dial=NUM where NUM is from 0 to
        100. it allows to choose percentage of tests to run. If you want
        stable set, you should pass --tester-random-seed=0 (or other
        integer). It is the first sketch and we have plans to improve
        it, but feedback would be welcome.<br>
        <br>
        > Is it needed on the tester?<br>
        <br>
        It is hard to say if it is strictly required for simple tests.
        However, it is better to update Tester as well, since
        performance tests run DPDK on Tester as well.<br>
        <br>
        > Are there any other manual setup steps for these devices
        that I might be missing?<br>
        <br>
        I don't remember anything else.<br>
        <br>
        I think it is better to get down to details and take a look at
        logs. I'm ready to help with it and explain what's happening
        there. May be it will help to understand if it is a problem with
        setup/configuration.<br>
        <br>
        Text logs are not very convenient. Ideally logs should be
        imported to bublik, however, manual runs do not provide all
        required artifacts right now (Jenkins jobs generate all required
        artifacts).<br>
        Other option is 'tmp_raw_log' file (should be packed to make it
        smaller) which could be converted to various log formats.<br>
        Would it be OK for you if I import your logs to bublik at
        ts-factory.io? Or is it a problem that it is publicly available?<br>
        Would it help if we add authentication and access control there?<br>
        <br>
        Andrew.<br>
        <br>
        On 9/8/23 17:57, Adam Hassick wrote:<br>
      </div>
      <blockquote type="cite"
cite="mid:CAC-YWqjOhiYXkBAK5pLnMqUNkB-z4Y=JAUMeN3fc6AMfjOzPrg@mail.gmail.com">
        <meta http-equiv="content-type" content="text/html;
          charset=UTF-8">
        <div dir="ltr">
          <div>
            <div>
              <div>
                <div>
                  <div>Hi Andrew,<br>
                    <br>
                  </div>
                  I have a couple questions about needed setup of the
                  NICs for the ethdev test suite.<br>
                  <br>
                </div>
                Our MCX5s and XL710s are failing the checkup tests. The
                pass rate appears to be much worse on the XL710s (40 of
                73 tests failed, 3 passed unexpectedly).<br>
                <br>
              </div>
              For the XL710s, I've updated the driver and NVM versions
              to match the minimum supported versions in the
              compatibility matrix found on the DPDK documentation. This
              did not change the failure rate much.<br>
            </div>
            For the MCX5s, I've installed the latest LTS version of the
            OFED bifurcated driver on the DUT. Is it needed on the
            tester?<br>
            <br>
          </div>
          Are there any other manual setup steps for these devices that
          I might be missing?<br>
          <div>
            <div>
              <div>
                <div>
                  <div>
                    <div>
                      <div><br>
                      </div>
                      <div>Thanks,<br>
                      </div>
                      <div>Adam<br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
        <br>
        <div class="gmail_quote">
          <div dir="ltr" class="gmail_attr">On Wed, Sep 6, 2023 at
            11:00 AM Adam Hassick <<a
              href="mailto:ahassick@iol.unh.edu" moz-do-not-send="true"
              class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>>
            wrote:<br>
          </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
            0.8ex;border-left:1px solid
            rgb(204,204,204);padding-left:1ex">
            <div dir="ltr">
              <div>
                <div>
                  <div>
                    <div>
                      <div>Hi Andrew,<br>
                        <br>
                      </div>
                      <div>Yes, I copied the X710 configs to set up
                        XL710 configs. I changed the environment
                        variable names from the X710 suffix to XL710
                        suffix in the script, and forgot to change them
                        in the corresponding environment file.<br>
                      </div>
                    </div>
                    That fixed the issue.<br>
                    <br>
                  </div>
                  I got the checkup tests working on the XL710 now. Most
                  of them are failing, which leads me to believe this is
                  an issue with our testbed. Based on the DPDK
                  documentation for i40e, the firmware and driver
                  versions are much older than what DPDK 22.11 LTS and
                  main prefer, so I'll try updating those.<br>
                  <br>
                </div>
                For now I'm working on getting the XL710 checkup tests
                passing, and will pick up getting the E810 configured
                properly next. I'll let you know if I run into any more
                issues in relation to the test engine.<br>
                <br>
              </div>
              <div>Thanks,<br>
              </div>
              <div>Adam<br>
              </div>
            </div>
            <br>
            <div class="gmail_quote">
              <div dir="ltr" class="gmail_attr">On Wed, Sep 6, 2023 at
                7:36 AM Andrew Rybchenko <<a
                  href="mailto:andrew.rybchenko@oktetlabs.ru"
                  target="_blank" moz-do-not-send="true"
                  class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
                wrote:<br>
              </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <div>
                  <div>Hi Adam,<br>
                    <br>
                    On 9/5/23 18:01, Adam Hassick wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div>
                        <div>
                          <div>
                            <div>
                              <div>
                                <div>Hi Andrew,<br>
                                  <br>
                                </div>
                                The compilation warning issue is now
                                resolved. Again, thank you guys for
                                fixing this for us. I can run the tests
                                on the Mellanox CX5s again, however I'm
                                running into a couple new issues with
                                running the prologues on the Intel
                                cards.<br>
                                <br>
                              </div>
                              When running testing on the Intel XL710s,
                              I see this error appear in the log:<br>
                              <br>
                              <blockquote class="gmail_quote"
                                style="margin:0px 0px 0px
                                0.8ex;border-left:1px solid
                                rgb(204,204,204);padding-left:1ex">ERROR
                                 prologue  Environment LIB  14:16:13.650<br>
                                Too few networks in available
                                configuration (0) in comparison with
                                required (1)<br>
                              </blockquote>
                              <br>
                            </div>
                            This seems like a trivial configuration
                            error, perhaps this is something I need to
                            set up in ts-rigs. I briefly searched
                            through the examples there and didn't see
                            any mention of how to set up a network.<br>
                          </div>
                          <div>I will attach this log just in case you
                            need more information.<br>
                          </div>
                        </div>
                      </div>
                    </div>
                  </blockquote>
                  <br>
                  Unfortunately logs are insufficient to understand it.
                  I've pushed new tag to TE v1.19.0 which add log
                  message with TE_* environment variables.<br>
                  Most likely something is wrong with variables which
                  are used as conditions when available networks are
                  defined in ts-conf/cs/inc.net_cfg_pci_fns.yml:<br>
                  TE_PCI_INSTANCE_IUT_TST1<br>
                  TE_PCI_INSTANCE_IUT_TST1a<br>
                  TE_PCI_INSTANCE_TST1a_IUT<br>
                  TE_PCI_INSTANCE_TST1_IUT<br>
                  My guess it that you change naming a bit, but script
                  like ts-rigs-sample/scripts/iut.h1-x710 is not
                  included or not updated.<br>
                  <br>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div>
                        <div>There is a different error when running on
                          the Intel E810s. It appears to me like it
                          starts DPDK, does some configuration inside
                          DPDK and on the device, and then fails to
                          bring the device back up. Since this error
                          seems very non-trivial, I will also attach
                          this log.<br>
                        </div>
                      </div>
                    </div>
                  </blockquote>
                  <br>
                  This one is a bit simpler. Few lines after the first
                  ERROR in log I see the following:<br>
                  WARN  RCF  DPDK  13:06:00.144<br>
                  ice_program_hw_rx_queue(): currently package doesn't
                  support RXDID (22)<br>
                  ice_rx_queue_start(): fail to program RX queue 0<br>
                  ice_dev_start(): fail to start Rx queue 0<br>
                  Device with port_id=0 already stopped<br>
                  <br>
                  It is stdout/stderr from test agent which runs DPDK.
                  Same logs in plain format are available in ta.DPDK
                  file.<br>
                  I'm not an expert here, but I vaguely remember that
                  E810 requires correct firmware and DDP to be loaded.<br>
                  There is some information in
                  dpdk/doc/guides/nics/ice.rst.<br>
                  <br>
                  You can try to add --dev-args=safe-mode-support=1
                  command-line option described there.<br>
                  <br>
                  Hope it helps,<br>
                  Andrew.<br>
                  <br>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div>
                        <div><br>
                        </div>
                        Thanks,<br>
                      </div>
                      Adam<br>
                    </div>
                    <br>
                    <div class="gmail_quote">
                      <div dir="ltr" class="gmail_attr">On Fri, Sep 1,
                        2023 at 3:59 AM Andrew Rybchenko <<a
                          href="mailto:andrew.rybchenko@oktetlabs.ru"
                          target="_blank" moz-do-not-send="true"
                          class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
                        wrote:<br>
                      </div>
                      <blockquote class="gmail_quote" style="margin:0px
                        0px 0px 0.8ex;border-left:1px solid
                        rgb(204,204,204);padding-left:1ex">
                        <div>
                          <div>Hi Adam,<br>
                            <br>
                            On 8/31/23 22:38, Adam Hassick wrote:<br>
                          </div>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div>Hi Andrew,<br>
                              </div>
                              <div><br>
                                I have one additional question as well:
                                Does the test engine support running
                                tests on two ARMv8 test agents?</div>
                              <div><br>
                              </div>
                              <div>
                                <blockquote class="gmail_quote"
                                  style="margin:0px 0px 0px
                                  0.8ex;border-left:1px solid
                                  rgb(204,204,204);padding-left:1ex">1.
                                  We'll sort out warnings this week.
                                  Thanks for heads up.<br>
                                </blockquote>
                                <div><br>
                                </div>
                                <div>Great. Let me know when that's
                                  fixed.</div>
                              </div>
                            </div>
                          </blockquote>
                          <br>
                          Done. We also fixed a number of warnings in
                          TE.<br>
                          Also we fixed root test package name to be
                          consistent with the repository name.<br>
                          <br>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <blockquote class="gmail_quote"
                                style="margin:0px 0px 0px
                                0.8ex;border-left:1px solid
                                rgb(204,204,204);padding-left:1ex">
                                <div>Support for old LTS branches was
                                  dropped some time ago, but in the
                                  future it is definitely possible to
                                  keep it for new LTS branches. I think
                                  22.11 is supported, but I'm not sure
                                  about older LTS releases.</div>
                              </blockquote>
                              <div><br>
                              </div>
                              <div>Good to know.<br>
                                <div> <br>
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex">
                                    2. You can add command-line option
                                    --sanity to run tests marked with
                                    TEST_HARNESS_SANITY requirement (see
                                    dpdk-ethdev-ts/scripts/run.sh and
                                    grep TEST_HARNESS_SANITY
                                    dpdk-ethdev-ts to see which tests
                                    are marked). Yes, there is a space
                                    for terminology improvement here.
                                    We'll do it.<br>
                                  </blockquote>
                                </div>
                              </div>
                            </div>
                          </blockquote>
                          <br>
                          Done. Now it is called --checkup.<br>
                          <br>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div>
                                <div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex"><br>
                                    Also it takes a lot of time because
                                    of failures and tests which wait for
                                    some timeout.<br>
                                  </blockquote>
                                </div>
                                <div><br>
                                </div>
                                <div>That makes sense to me. We'll use
                                  the time to complete tests on virtio
                                  or the Intel devices as a reference
                                  for how long the tests really take to
                                  complete.<br>
                                </div>
                                <div>We will explore the possibility of
                                  periodically running the sanity tests
                                  for patches.<br>
                                </div>
                              </div>
                            </div>
                          </blockquote>
                          <br>
                          I'll double-check and let you know how long
                          entire TS runs on Intel X710, E810, Mellanox
                          CX5 and virtio net. Just to ensure that time
                          observed in your case looks the same.<br>
                          <br>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div>
                                <div> <br>
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex">
                                    The test harness can provide
                                    coverage reports based on gcov, but
                                    I'm not sure what you mean by a
                                    "dial" to control test coverage.
                                    Provided reports are rather for
                                    human to analyze.<br>
                                  </blockquote>
                                </div>
                                <div><br>
                                </div>
                                <div>The general idea is to have some
                                  kind of parameter on the test suite,
                                  which could be an integer ranging from
                                  zero to ten, that controls how many
                                  tests are run based on how important
                                  the test is.<br>
                                  <br>
                                </div>
                                <div>Similar to how some command line
                                  interfaces provide a verbosity level
                                  parameter (some number of "-v"
                                  arguments) to control the importance
                                  of the information in the log.<br>
                                </div>
                                The verbosity level zero only prints
                                very important log messages, while ten
                                prints everything.<br>
                              </div>
                              <div><br>
                                In much the same manner as above, this
                                "dial" parameter controls what tests are
                                run and with what parameters based on
                                how important those tests and test
                                parameter combinations are.<br>
                                Coverage Level zero tells the suite to
                                run a very basic set of important tests,
                                with minimal parameterization. This mode
                                would take only ~5-10 minutes to run.<br>
                                In contrast, Coverage Level ten includes
                                all the edge cases, every combination of
                                test parameters, everything the test
                                suite can do, which takes the normal
                                several hours to run.<br>
                                The values 1 - 9 are between those two
                                extremes, allowing the user to get a
                                gradient of test coverage in the results
                                and to limit the running time.<br>
                                <br>
                              </div>
                              Then we could, for example, run the
                              "run.sh" with a level of 2 or 3 for
                              incoming patches that need quick results,
                              and with a level of 10 for the less often
                              run periodic tests performed on main or
                              LTS branches.<br>
                            </div>
                          </blockquote>
                          <br>
                          Understood now. Thanks a lot for the idea.
                          We'll discuss it and come back.<br>
                          <br>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div>
                                <div>
                                  <div> </div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex">
                                    <div>
                                      <div> 3. Yes, really many tests on
                                        Mellanox CX5 NICs report
                                        unexpected testing results.
                                        Unfortunately it is time
                                        consuming to fill in
                                        expectations database since it
                                        is necessary to analyze testing
                                        results and classify if it is a
                                        bug or just acceptable behaviour
                                        aspect.<br>
                                        <br>
                                        Bublik allows to compare results
                                        of two runs. It is useful for
                                        human, but still not good for
                                        automation.<br>
                                        <br>
                                        I have local patch for mlx5
                                        driver which reports Tx ring
                                        size maximum. It makes pass rate
                                        higher. It is a problem for test
                                        harness that mlx5 does not
                                        report limits right now.<br>
                                        <br>
                                        Pass rate on Intel X710 is about
                                        92% on my test rig. Pass rate on
                                        virtio net is 99% right now and
                                        could be done 100% easily (just
                                        one thing to fix in
                                        expectations).<br>
                                        <br>
                                        I think logs storage setup is
                                        essential for logs analysis. Of
                                        course, you can request HTML
                                        logs when you run tests
                                        (--log-html=html) or generate
                                        after run using
                                        dpdk-ethdev-ts/scripts/html-log.sh
                                        and open index.html in a
                                        browser, but logs storage makes
                                        it more convenient.<br>
                                      </div>
                                    </div>
                                  </blockquote>
                                  <div><br>
                                    We are interested in setting up
                                    Bublik, potentially as an
                                    externally-facing component, once we
                                    have our process of running the test
                                    suite stabilized.</div>
                                  <div>Once we are able to run the test
                                    suite again, I'll see what the pass
                                    rate is on our other hardware.<br>
                                    Good to know that it isn't an issue
                                    with our dev testbed causing the
                                    high fail rate.</div>
                                </div>
                                <div>
                                  <div><br>
                                  </div>
                                  <div>For Intel hardware, we have an
                                    XL710 and an Intel E810-C in our
                                    development testbed. Although they
                                    are slightly different devices,
                                    ideally the pass rate will be
                                    identical or similar. I have yet to
                                    set up a VM pair for virtio, but we
                                    will soon.<br>
                                  </div>
                                  <div><br>
                                    <blockquote class="gmail_quote"
                                      style="margin:0px 0px 0px
                                      0.8ex;border-left:1px solid
                                      rgb(204,204,204);padding-left:1ex">
                                      Latest version of test-environment
                                      has examples of our CGI scripts
                                      which we use for log storage (see
                                      tools/log_server/README.md).<br>
                                      <br>
                                      Also all bits for Jenkins setup
                                      are available. See
                                      dpdk-ethdev-ts/jenkins/README.md
                                      and examples of jenkins files in
                                      ts-rigs-sample.<br>
                                    </blockquote>
                                  </div>
                                  <div><br>
                                  </div>
                                  <div>Jenkins integration, setting up
                                    production rig configurations, and
                                    permanent log storage will be our
                                    next steps once I am able to run the
                                    tests again.<br>
                                  </div>
                                  <div>Unless there is an easy way to
                                    have meson not pass "-Werror" into
                                    GCC. Then I would be able to run the
                                    test suite.<br>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </blockquote>
                          <br>
                          Hopefully it is resolved now.<br>
                          <br>
                          I thought a bit more about your usecase for
                          Jenkins. I'm not 100% sure that existing
                          pipelines are convenient for your usecase.<br>
                          Fill free to ask questions when you are on it.<br>
                          <br>
                          Thanks,<br>
                          Andrew.<br>
                          <br>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div>
                                <div>
                                  <div><br>
                                  </div>
                                  <div>Thanks,<br>
                                  </div>
                                  <div>Adam<br>
                                  </div>
                                  <div><br>
                                  </div>
                                  <div> </div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex">
                                    <div>
                                      <div> <br>
                                        On 8/29/23 17:02, Adam Hassick
                                        wrote:<br>
                                      </div>
                                      <blockquote type="cite">
                                        <div dir="ltr">
                                          <div>
                                            <div>
                                              <div>Hi Andrew,<br>
                                                <br>
                                              </div>
                                              That fix seems to have
                                              resolved the issue, thanks
                                              for the quick turnaround
                                              time on that patch.<br>
                                            </div>
                                            <div>Now that we have the
                                              RCF timeout issue
                                              resolved, there are a few
                                              other questions and issues
                                              that we have about the
                                              tests themselves.</div>
                                            <br>
                                          </div>
                                          <div>1. The test suite fails
                                            to build with a couple
                                            warnings.<br>
                                          </div>
                                          <div><br>
                                          </div>
                                          <div>Below is the stderr log
                                            from compilation:<br>
                                          </div>
                                          <br>
                                          <blockquote
                                            class="gmail_quote"
                                            style="margin:0px 0px 0px
                                            0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">FAILED: <a
                                              href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                              target="_blank"
                                              moz-do-not-send="true"
                                              class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
                                            cc
                                            -Ilib/76b5a35@@ts_dpdk_pmd@sta
                                            -Ilib -I../../lib
                                            -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
                                            -fdiagnostics-color=always
                                            -pipe -D_FILE_OFFSET_BITS=64
                                            -Wall -Winvalid-pch -Werror
                                            -g -D_GNU_SOURCE -O0 -ggdb
                                            -Wall -W -fPIC -MD -MQ '<a
                                              href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                              target="_blank"
                                              moz-do-not-send="true"
                                              class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
                                            -MF '<a
                                              href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d"
                                              target="_blank"
                                              moz-do-not-send="true"
                                              class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
                                            -o '<a
                                              href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                              target="_blank"
                                              moz-do-not-send="true"
                                              class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
                                            -c ../../lib/dpdk_pmd_ts.c<br>
                                            ../../lib/dpdk_pmd_ts.c: In
                                            function
                                            ‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
                                            no format arguments
                                            [-Werror=format-security]<br>
                                            5577 |     rc =
                                            te_kvpair_add(result, buf,
                                            mode);<br>
                                            |     ^~<br>
                                            cc1: all warnings being
                                            treated as errors<br>
                                            ninja: build stopped:
                                            subcommand failed.<br>
                                            ninja: Entering directory
                                            `.'<br>
                                            FAILED: <a
                                              href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                              target="_blank"
                                              moz-do-not-send="true"
                                              class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
                                            cc
                                            -Ilib/76b5a35@@ts_dpdk_pmd@sta
                                            -Ilib -I../../lib
                                            -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
                                            -fdiagnostics-color=always
                                            -pipe -D_FILE_OFFSET_BITS=64
                                            -Wall -Winvalid-pch -Werror
                                            -g -D_GNU_SOURCE -O0 -ggdb
                                            -Wall -W -fPIC -MD -MQ '<a
                                              href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                              target="_blank"
                                              moz-do-not-send="true"
                                              class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
                                            -MF '<a
                                              href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d"
                                              target="_blank"
                                              moz-do-not-send="true"
                                              class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d</a>'
                                            -o '<a
                                              href="mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o"
                                              target="_blank"
                                              moz-do-not-send="true"
                                              class="moz-txt-link-freetext">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>'
                                            -c ../../lib/dpdk_pmd_ts.c<br>
                                            ../../lib/dpdk_pmd_ts.c: In
                                            function
                                            ‘test_create_traffic_generator_params’:<br>
../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and
                                            no format arguments
                                            [-Werror=format-security]<br>
                                            5577 |     rc =
                                            te_kvpair_add(result, buf,
                                            mode);<br>
                                            |     ^~<br>
                                            cc1: all warnings being
                                            treated as errors<br>
                                          </blockquote>
                                          <div>
                                            <div>
                                              <div><br>
                                              </div>
                                              <div>This error wasn't
                                                occurring last week,
                                                which was the last time
                                                I ran the tests.<br>
                                              </div>
                                              <div>The TE host and the
                                                DUT have GCC v9.4.0
                                                installed, and the
                                                tester has GCC v11.4.0
                                                installed, if this
                                                information is helpful.<br>
                                              </div>
                                              <div><br>
                                              </div>
                                              <div>2. On the Mellanox
                                                CX5s, there are over
                                                6,000 tests run, which
                                                collectively take around
                                                9 hours. Is it possible,
                                                and would it make sense,
                                                to lower the test
                                                coverage and have the
                                                test suite run faster?<br>
                                                <br>
                                              </div>
                                              <div>For some context, we
                                                run immediate testing on
                                                incoming patches for
                                                DPDK main and
                                                development branches, as
                                                well as periodic test
                                                runs on the main,
                                                stable, and LTS
                                                branches.<br>
                                              </div>
                                              <div>For us to consider
                                                including this test
                                                suite as part of our
                                                immediate testing on
                                                patches, we would have
                                                to reduce the test
                                                coverage to the most
                                                important tests.<br>
                                                This is primarily to
                                                reduce the testing time
                                                to, for example, less
                                                than 30 minutes. Testing
                                                on patches can't take
                                                too long because the lab
                                                can receive numerous
                                                patches each day, which
                                                each require individual
                                                testing runs.<br>
                                                <br>
                                              </div>
                                              <div>At what frequency we
                                                run these tests, and on
                                                what, still needs to be
                                                discussed with the DPDK
                                                community, but it would
                                                be nice to know if the
                                                test suite had a "dial"
                                                to control the testing
                                                coverage.<br>
                                              </div>
                                              <div><br>
                                              </div>
                                              <div>3. We see a lot of
                                                test failures on our
                                                Mellanox CX5 NICs.
                                                Around 2,300 of ~6,600
                                                tests passed. Is there
                                                anything we can do to
                                                diagnose these test
                                                failures?<br>
                                              </div>
                                              <div><br>
                                              </div>
                                              <div>Thanks,<br>
                                              </div>
                                              <div>Adam<br>
                                              </div>
                                              <div><br>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                        <br>
                                        <div class="gmail_quote">
                                          <div dir="ltr"
                                            class="gmail_attr">On Tue,
                                            Aug 29, 2023 at 8:07 AM
                                            Andrew Rybchenko <<a
                                              href="mailto:andrew.rybchenko@oktetlabs.ru"
                                              target="_blank"
                                              moz-do-not-send="true"
                                              class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
                                            wrote:<br>
                                          </div>
                                          <blockquote
                                            class="gmail_quote"
                                            style="margin:0px 0px 0px
                                            0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
                                            <div>
                                              <div>Hi Adam,<br>
                                                <br>
                                                I've pushed the fix in
                                                main branch and a new
                                                tag v1.18.1. It should
                                                solve the problem with
                                                IPv6 address from DNS.<br>
                                                <br>
                                                Andrew.<br>
                                                <br>
                                                On 8/29/23 00:05, Andrew
                                                Rybchenko wrote:<br>
                                              </div>
                                              <blockquote type="cite">
                                                <div>Hi Adam,<br>
                                                  <br>
                                                  > Does the test
                                                  engine prefer to use
                                                  IPv6 over IPv4 for
                                                  initiating the RCF
                                                  connection to the test
                                                  bed hosts? And if so,
                                                  is there a way to
                                                  force it to use IPv4?<br>
                                                  <br>
                                                  Brilliant idea. If DNS
                                                  returns both IPv4 and
                                                  IPv6 addresses in your
                                                  case, I guess it is
                                                  the root cause of the
                                                  problem.<br>
                                                  Of course, it is TE
                                                  problem since I see
                                                  really weird code in
                                                  lib/comm_net_engine/comm_net_engine.c
                                                  line 135.<br>
                                                  <br>
                                                  I've pushed fix to the
                                                  branch
                                                  user/arybchik/fix_ipv4_only
                                                  in
                                                  ts-factory/test-environment
                                                  repository. Please,
                                                  try.<br>
                                                  <br>
                                                  It is late night fix
                                                  with minimal testing
                                                  and no review. I'll
                                                  pass it through review
                                                  process tomorrow and<br>
                                                  hopefully it will be
                                                  released in one-two
                                                  days.<br>
                                                  <br>
                                                  Andrew.<br>
                                                  <br>
                                                  On 8/28/23 18:02, Adam
                                                  Hassick wrote:<br>
                                                </div>
                                                <blockquote type="cite">
                                                  <div dir="ltr">
                                                    <div>
                                                      <div>
                                                        <div>Hi Andrew,<br>
                                                          <br>
                                                        </div>
                                                        We have yet to
                                                        notice a
                                                        distinct pattern
                                                        with the
                                                        failures.
                                                        Sometimes, the
                                                        RCF will start
                                                        and connect
                                                        without issue a
                                                        few times in a
                                                        row before
                                                        failing to
                                                        connect again.
                                                        Once the issue
                                                        begins to occur,
                                                        neither
                                                        rebooting all of
                                                        the hosts (test
                                                        engine VM,
                                                        tester, IUT) or
                                                        deleting all of
                                                        the build
                                                        directories
                                                        (suites, agents,
                                                        inst) and
                                                        rebooting the
                                                        hosts afterward
                                                        resolves the
                                                        issue. When it
                                                        begins working
                                                        again seems very
                                                        arbitrary to us.<br>
                                                        <br>
                                                      </div>
                                                      <div>I do usually
                                                        try to terminate
                                                        the test engine
                                                        with Ctrl+C, but
                                                        when it hangs
                                                        while trying to
                                                        start RCF, that
                                                        does not work.<br>
                                                      </div>
                                                      <div><br>
                                                      </div>
                                                      <div>Does the test
                                                        engine prefer to
                                                        use IPv6 over
                                                        IPv4 for
                                                        initiating the
                                                        RCF connection
                                                        to the test bed
                                                        hosts? And if
                                                        so, is there a
                                                        way to force it
                                                        to use IPv4?<br>
                                                        <br>
                                                      </div>
                                                      <div> - Adam<br>
                                                      </div>
                                                    </div>
                                                  </div>
                                                  <br>
                                                  <div
                                                    class="gmail_quote">
                                                    <div dir="ltr"
                                                      class="gmail_attr">On
                                                      Fri, Aug 25, 2023
                                                      at 1:35 PM Andrew
                                                      Rybchenko <<a
                                                        href="mailto:andrew.rybchenko@oktetlabs.ru"
                                                        target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
                                                      wrote:<br>
                                                    </div>
                                                    <blockquote
                                                      class="gmail_quote"
                                                      style="margin:0px
                                                      0px 0px
                                                      0.8ex;border-left:1px
                                                      solid
                                                      rgb(204,204,204);padding-left:1ex">
                                                      <div>
                                                        <div>> I'll
                                                          double-check
                                                          test engine on
                                                          Ubuntu 20.04
                                                          and Ubuntu
                                                          22.04.<br>
                                                          <br>
                                                          Done. It works
                                                          fine for me
                                                          without any
                                                          issues.<br>
                                                          <br>
                                                          Have you
                                                          noticed any
                                                          pattern when
                                                          it works or
                                                          does not work?<br>
                                                          May be it is a
                                                          problem of not
                                                          clean state
                                                          after
                                                          termination?<br>
                                                          Does it work
                                                          fine the first
                                                          time after
                                                          DUTs reboot?<br>
                                                          How do you
                                                          terminate
                                                          testing? It
                                                          should be done
                                                          using Ctrl+C
                                                          in terminal
                                                          where you
                                                          execute run.sh
                                                          command.<br>
                                                           In this case
                                                          it should
                                                          shutdown
                                                          gracefully and
                                                          close all test
                                                          agents and
                                                          engine
                                                          applications.<br>
                                                          <br>
                                                          (I'm trying to
                                                          understand why
                                                          you've seen
                                                          many test
                                                          agent
                                                          processes. It
                                                          should not
                                                          happen.)<br>
                                                          <br>
                                                          Andrew.<br>
                                                          <br>
                                                          On 8/25/23
                                                          17:41, Andrew
                                                          Rybchenko
                                                          wrote:<br>
                                                        </div>
                                                        <blockquote
                                                          type="cite">
                                                          <div>On
                                                          8/25/23 17:06,
                                                          Adam Hassick
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>Hi
                                                          Andrew,<br>
                                                          <br>
                                                          </div>
                                                          Two of our
                                                          systems (the
                                                          Test Engine
                                                          runner and the
                                                          DUT host) are
                                                          running Ubuntu
                                                          20.04 LTS,
                                                          however this
                                                          morning I
                                                          noticed that
                                                          the tester
                                                          system (the
                                                          one having
                                                          issues) is
                                                          running Ubuntu
                                                          22.04 LTS.<br>
                                                          </div>
                                                          <div>This
                                                          could be the
                                                          source of the
                                                          problem. I
                                                          encountered a
                                                          dependency
                                                          issue trying
                                                          to run the
                                                          Test Engine on
                                                          22.04 LTS, so
                                                          I downgraded
                                                          the system.
                                                          Since the
                                                          tester is also
                                                          the host
                                                          having
                                                          connection
                                                          issues, I will
                                                          try
                                                          downgrading
                                                          that system to
                                                          20.04, and see
                                                          if that
                                                          changes
                                                          anything.<br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          Unlikely, but
                                                          who knows. We
                                                          run tests
                                                          (DUTs) on
                                                          Ubuntu 20.04,
                                                          Ubuntu 22.04,
                                                          Ubuntu 22.10,
                                                          Ubuntu 23.04,
                                                          Debian 11 and
                                                          Fedora 38
                                                          every night.<br>
                                                          Right now
                                                          Debian 11 is
                                                          used for test
                                                          engine in
                                                          nightly
                                                          regressions.<br>
                                                          <br>
                                                          I'll
                                                          double-check
                                                          test engine on
                                                          Ubuntu 20.04
                                                          and Ubuntu
                                                          22.04.<br>
                                                          <br>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div>I did try
                                                          passing in the
                                                          "--vg-rcf"
                                                          argument to
                                                          the run.sh
                                                          script of the
                                                          test suite
                                                          after
                                                          installing
                                                          valgrind, but
                                                          there was no
                                                          additional
                                                          output that I
                                                          saw.<br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          Sorry, I
                                                          should
                                                          valgrind
                                                          output should
                                                          be in
                                                          valgrind.te_rcf
                                                          (direction
                                                          where you run
                                                          test engine).<br>
                                                          <br>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div><br>
                                                          </div>
                                                          <div>I will
                                                          try pulling in
                                                          the changes
                                                          you've pushed
                                                          up, and will
                                                          see if that
                                                          fixes
                                                          anything.<br>
                                                          <br>
                                                          </div>
                                                          <div>Thanks,<br>
                                                          </div>
                                                          <div>Adam<br>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">
                                                          <div dir="ltr"
class="gmail_attr">On Fri, Aug 25, 2023 at 9:57 AM Andrew Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div>
                                                          <div>Hello
                                                          Adam, <br>
                                                          <br>
                                                          On 8/24/23
                                                          23:54, Andrew
                                                          Rybchenko
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">I'd
                                                          like to try to
                                                          repeat the
                                                          problem
                                                          locally. Which
                                                          Linux distro
                                                          is running on
                                                          test engine
                                                          and agents? <br>
                                                          <br>
                                                          In fact I know
                                                          one problem
                                                          with Debian 12
                                                          and Fedora 38
                                                          and we have <br>
                                                          patch in
                                                          review to fix
                                                          it, however,
                                                          the behaviour
                                                          is different
                                                          in <br>
                                                          this case, so
                                                          it is unlike
                                                          the same
                                                          problem. <br>
                                                          </blockquote>
                                                          <br>
                                                          I've just
                                                          published a
                                                          new tag which
                                                          fixes known
                                                          test engine
                                                          side problems
                                                          on Debian 12
                                                          and Fedora 38.<br>
                                                          <br>
                                                          <blockquote
                                                          type="cite"> <br>
                                                          One more idea
                                                          is to install
                                                          valgrind on
                                                          the test
                                                          engine host
                                                          and <br>
                                                          run with
                                                          option
                                                          --vg-rcf to
                                                          check if
                                                          something
                                                          weird is
                                                          happening. <br>
                                                          <br>
                                                          What I don't
                                                          understand
                                                          right now is
                                                          why I see just
                                                          one failed
                                                          attempt <br>
                                                          to connect in
                                                          your log.txt
                                                          and then
                                                          Logger
                                                          shutdown after
                                                          9 minutes. <br>
                                                          <br>
                                                          Andrew. <br>
                                                          <br>
                                                          On 8/24/23
                                                          23:29, Adam
                                                          Hassick wrote:
                                                          <br>
                                                          <blockquote
                                                          type="cite"> >
                                                          Is there any
                                                          firewall in
                                                          the network or
                                                          on test hosts
                                                          which could
                                                          block incoming
                                                          TCP connection
                                                          to the port
                                                          23571 <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          from the host
                                                          where you run
                                                          test engine? <br>
                                                          <br>
                                                          Our test
                                                          engine host
                                                          and the
                                                          testbed are on
                                                          the same
                                                          subnet. The
                                                          connection
                                                          does work
                                                          sometimes. <br>
                                                          <br>
                                                           > If
                                                          behaviour the
                                                          same on the
                                                          next try and
                                                          you see that
                                                          test agent is
                                                          kept running,
                                                          could you
                                                          check using <br>
                                                           > <br>
                                                           > #
                                                          netstat -tnlp
                                                          <br>
                                                           > <br>
                                                           > that
                                                          Test Agent is
                                                          listening on
                                                          the port and
                                                          try to
                                                          establish TCP
                                                          connection
                                                          from test
                                                          agent using <br>
                                                           > <br>
                                                           > $ telnet
                                                          <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu"
target="_blank" moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a>
                                                          <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          23571 <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          <br>
                                                           > <br>
                                                           > and
                                                          check if TCP
                                                          connection
                                                          could be
                                                          established. <br>
                                                          <br>
                                                          I was able to
                                                          replicate the
                                                          same behavior
                                                          again, where
                                                          it hangs while
                                                          RCF is trying
                                                          to start. <br>
                                                          Running this
                                                          command, I see
                                                          this in the
                                                          output: <br>
                                                          <br>
                                                          tcp        0  
                                                             0 <a
                                                          href="http://0.0.0.0:23571"
target="_blank" moz-do-not-send="true">0.0.0.0:23571</a> <a
                                                          href="http://0.0.0.0:23571"
target="_blank" moz-do-not-send="true"><http://0.0.0.0:23571></a>          
                                                          0.0.0.0:*    
                                                                   
                                                          LISTEN    
                                                           18599/ta <br>
                                                          <br>
                                                          So it seems
                                                          like it is
                                                          listening on
                                                          the correct
                                                          port. <br>
                                                          Additionally,
                                                          I was able to
                                                          connect to the
                                                          Tester machine
                                                          from our Test
                                                          Engine host
                                                          using telnet.
                                                          It printed the
                                                          PID of the
                                                          process once
                                                          the connection
                                                          was opened. <br>
                                                          <br>
                                                          I tried
                                                          running the
                                                          "ta"
                                                          application
                                                          manually on
                                                          the command
                                                          line, and it
                                                          didn't print
                                                          anything at
                                                          all. <br>
                                                          Maybe the
                                                          issue is
                                                          something on
                                                          the Test
                                                          Engine side. <br>
                                                          <br>
                                                          On Thu, Aug
                                                          24, 2023 at
                                                          2:35 PM Andrew
                                                          Rybchenko <<a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a> <a
                                                          href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
                                                          wrote: <br>
                                                          <br>
                                                              Hi Adam, <br>
                                                          <br>
                                                               > On
                                                          the tester
                                                          host (which
                                                          appears to be
                                                          the Peer
                                                          agent), there
                                                          <br>
                                                              are four
                                                          processes that
                                                          I see running,
                                                          which look
                                                          like the test
                                                          <br>
                                                              agent
                                                          processes. <br>
                                                          <br>
                                                              Before the
                                                          next try I'd
                                                          recommend to
                                                          kill these
                                                          processes. <br>
                                                          <br>
                                                              Is there
                                                          any firewall
                                                          in the network
                                                          or on test
                                                          hosts which
                                                          could <br>
                                                              block
                                                          incoming TCP
                                                          connection to
                                                          the port 23571
                                                          <br>
                                                              <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          from the host
                                                          <br>
                                                              where you
                                                          run test
                                                          engine? <br>
                                                          <br>
                                                              If
                                                          behaviour the
                                                          same on the
                                                          next try and
                                                          you see that
                                                          test agent is
                                                          <br>
                                                              kept
                                                          running, could
                                                          you check
                                                          using <br>
                                                          <br>
                                                              # netstat
                                                          -tnlp <br>
                                                          <br>
                                                              that Test
                                                          Agent is
                                                          listening on
                                                          the port and
                                                          try to
                                                          establish TCP
                                                          <br>
                                                              connection
                                                          from test
                                                          agent using <br>
                                                          <br>
                                                              $ telnet <a
href="http://iol-dts-tester.dpdklab.iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu</a>
                                                          <br>
                                                              <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          23571 <br>
                                                              <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          <br>
                                                          <br>
                                                              and check
                                                          if TCP
                                                          connection
                                                          could be
                                                          established. <br>
                                                          <br>
                                                              Another
                                                          idea is to
                                                          login Tester
                                                          under root as
                                                          testing does,
                                                          get <br>
                                                              start TA
                                                          command from
                                                          the log and
                                                          try it by
                                                          hands without
                                                          -n and <br>
                                                              remove
                                                          extra
                                                          escaping. <br>
                                                          <br>
                                                              # sudo
                                                          PATH=${PATH}:/tmp/linux_x86_root_76872_1692885663_1
                                                          <br>
                                                             
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_root_76872_1692885663_1
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=iol-dts-tester.dpdklab.iol.unh.edu:port=23571:user=root:key=/opt/tsf/keys/id_ed25519:ssh_port=22:copy_timeout=15:kill_timeout=15:sudo=:shell=<br>
                                                          <br>
                                                              Hopefully
                                                          in this case
                                                          test agent
                                                          directory
                                                          remains in the
                                                          /tmp and <br>
                                                              you don't
                                                          need to copy
                                                          it as testing
                                                          does. <br>
                                                              May be
                                                          output could
                                                          shed some
                                                          light on
                                                          what's going
                                                          on. <br>
                                                          <br>
                                                              Andrew. <br>
                                                          <br>
                                                              On 8/24/23
                                                          17:30, Adam
                                                          Hassick wrote:
                                                          <br>
                                                          <blockquote
                                                          type="cite">   
                                                          Hi Andrew, <br>
                                                          <br>
                                                              This is
                                                          the output
                                                          that I see in
                                                          the terminal
                                                          when this
                                                          failure <br>
                                                              occurs,
                                                          after the test
                                                          agent binaries
                                                          build and the
                                                          test engine <br>
                                                              starts: <br>
                                                          <br>
                                                              Platform
                                                          default build
                                                          - pass <br>
                                                              Simple RCF
                                                          consistency
                                                          check
                                                          succeeded <br>
                                                             
                                                          --->>>
                                                          Starting
                                                          Logger...done
                                                          <br>
                                                             
                                                          --->>>
                                                          Starting
                                                          RCF...rcf_net_engine_connect():
                                                          Connection
                                                          timed <br>
                                                              out <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true">iol-dts-tester.dpdklab.iol.unh.edu:23571</a>
                                                          <br>
                                                              <a
                                                          href="http://iol-dts-tester.dpdklab.iol.unh.edu:23571"
target="_blank" moz-do-not-send="true"><http://iol-dts-tester.dpdklab.iol.unh.edu:23571></a>
                                                          <br>
                                                          <br>
                                                              Then, it
                                                          hangs here
                                                          until I kill
                                                          the "te_rcf"
                                                          and "te_tee" <br>
                                                              processes.
                                                          I let it hang
                                                          for around 9
                                                          minutes. <br>
                                                          <br>
                                                              On the
                                                          tester host
                                                          (which appears
                                                          to be the Peer
                                                          agent), there
                                                          are <br>
                                                              four
                                                          processes that
                                                          I see running,
                                                          which look
                                                          like the test
                                                          agent <br>
                                                              processes.
                                                          <br>
                                                          <br>
                                                              ta.Peer is
                                                          an empty file.
                                                          I've attached
                                                          the log.txt
                                                          from this run.
                                                          <br>
                                                          <br>
                                                               - Adam <br>
                                                          <br>
                                                              On Thu,
                                                          Aug 24, 2023
                                                          at 4:22 AM
                                                          Andrew
                                                          Rybchenko <br>
                                                              <<a
                                                          href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
                                                          <br>
                                                              <a
                                                          href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
                                                          wrote: <br>
                                                          <br>
                                                                  Hi
                                                          Adam, <br>
                                                          <br>
                                                                  Yes,
                                                          TE_RCFUNIX_TIMEOUT
                                                          is in seconds.
                                                          I've
                                                          double-checked
                                                          <br>
                                                                  that
                                                          it goes to
                                                          'copy_timeout'
                                                          in
                                                          ts-conf/rcf.conf.
                                                          <br>
                                                                 
                                                          Description in
                                                          in
                                                          doc/sphinx/pages/group_te_engine_rcf.rst
                                                          <br>
                                                                  says
                                                          that
                                                          copy_timeout
                                                          is in seconds
                                                          and
                                                          implementation
                                                          in <br>
                                                                 
                                                          lib/rcfunix/rcfunix.c
                                                          passes the
                                                          value to
                                                          select()
                                                          tv_sec. <br>
                                                                 
                                                          Theoretically
                                                          select() could
                                                          be interrupted
                                                          by signal, but
                                                          I <br>
                                                                  think
                                                          it is unlikely
                                                          here. <br>
                                                          <br>
                                                                  I'm
                                                          not sure that
                                                          I understand
                                                          what do you
                                                          mean by RCF <br>
                                                                 
                                                          connection
                                                          timeout. Does
                                                          it happen on
                                                          TE startup
                                                          when RCF <br>
                                                                  starts
                                                          test agents.
                                                          If so,
                                                          TE_RCFUNIX_TIMEOUT
                                                          could help. Or
                                                          <br>
                                                                  does
                                                          it happen when
                                                          tests are in
                                                          progress, e.g.
                                                          in the middle
                                                          <br>
                                                                  of a
                                                          test. If so,
                                                          TE_RCFUNIX_TIMEOUT
                                                          is unrelated
                                                          and most <br>
                                                                  likely
                                                          either host
                                                          with test
                                                          agent dies or
                                                          test agent
                                                          itself <br>
                                                                 
                                                          crashes. It
                                                          would be
                                                          easier for me
                                                          if classify it
                                                          if you share <br>
                                                                  text
                                                          log (log.txt,
                                                          full or just
                                                          corresponding
                                                          fragment with
                                                          <br>
                                                                  some
                                                          context). Also
                                                          content of
                                                          ta.DPDK or
                                                          ta.Peer file <br>
                                                                 
                                                          depending on
                                                          which agent
                                                          has problems
                                                          could shed
                                                          some light. <br>
                                                                 
                                                          Corresponding
                                                          files contain
                                                          stdout/stderr
                                                          of test
                                                          agents. <br>
                                                          <br>
                                                                 
                                                          Andrew. <br>
                                                          <br>
                                                                  On
                                                          8/23/23 17:45,
                                                          Adam Hassick
                                                          wrote: <br>
                                                          <blockquote
                                                          type="cite">       
                                                          Hi Andrew, <br>
                                                          <br>
                                                                  I've
                                                          set up a test
                                                          rig repository
                                                          here, and have
                                                          created <br>
                                                                 
                                                          configurations
                                                          for our
                                                          development
                                                          testbed based
                                                          off of the <br>
                                                                 
                                                          examples. <br>
                                                                  We've
                                                          been able to
                                                          get the test
                                                          suite to run
                                                          manually on <br>
                                                                 
                                                          Mellanox CX5
                                                          devices once.
                                                          <br>
                                                                 
                                                          However, we
                                                          are running
                                                          into an issue
                                                          where, when
                                                          RCF starts, <br>
                                                                  the
                                                          RCF connection
                                                          times out very
                                                          frequently. We
                                                          aren't sure <br>
                                                                  why
                                                          this is the
                                                          case. <br>
                                                                  It
                                                          works
                                                          sometimes, but
                                                          most of the
                                                          time when we
                                                          try to run <br>
                                                                  the
                                                          test engine,
                                                          it encounters
                                                          this issue. <br>
                                                                  I've
                                                          tried changing
                                                          the RCF port
                                                          by setting <br>
                                                                 
                                                          "TE_RCF_PORT=<some
                                                          port
                                                          number>"
                                                          and rebooting
                                                          the testbed <br>
                                                                 
                                                          machines.
                                                          Neither seems
                                                          to fix the
                                                          issue. <br>
                                                          <br>
                                                                  It
                                                          also seems
                                                          like the
                                                          timeout takes
                                                          far longer
                                                          than 60 <br>
                                                                 
                                                          seconds, even
                                                          when running
                                                          "export
                                                          TE_RCFUNIX_TIMEOUT=60"
                                                          <br>
                                                                  before
                                                          I try to run
                                                          the test
                                                          suite. <br>
                                                                  I
                                                          assume the
                                                          unit for this
                                                          variable is
                                                          seconds? <br>
                                                          <br>
                                                                 
                                                          Thanks, <br>
                                                                  Adam <br>
                                                          <br>
                                                                  On
                                                          Mon, Aug 21,
                                                          2023 at
                                                          10:19 AM Adam
                                                          Hassick <br>
                                                                  <<a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a> <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>>
                                                          wrote: <br>
                                                          <br>
                                                                      Hi
                                                          Andrew, <br>
                                                          <br>
                                                                     
                                                          Thanks, I've
                                                          cloned the
                                                          example
                                                          repository and
                                                          will start <br>
                                                                     
                                                          setting up a
                                                          configuration
                                                          for our
                                                          development
                                                          testbed <br>
                                                                     
                                                          today. I'll
                                                          let you know
                                                          if I run into
                                                          any
                                                          difficulties <br>
                                                                      or
                                                          have any
                                                          questions. <br>
                                                          <br>
                                                                       -
                                                          Adam <br>
                                                          <br>
                                                                      On
                                                          Sun, Aug 20,
                                                          2023 at
                                                          4:40 AM Andrew
                                                          Rybchenko <br>
                                                                     
                                                          <<a
                                                          href="mailto:andrew.rybchenko@oktetlabs.ru"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
                                                          <br>
                                                                      <a
href="mailto:andrew.rybchenko@oktetlabs.ru" target="_blank"
                                                          moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
                                                          wrote: <br>
                                                          <br>
                Hi Adam, <br>
                                                          <br>
                I've published <br>
                <a href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ts-factory/ts-rigs-sample</a>
                                                          <br>
                <a href="https://github.com/ts-factory/ts-rigs-sample"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/ts-rigs-sample></a>.
                                                          <br>
                Hopefully it will help to define your test rigs and <br>
                successfully run some tests manually. Feel free to <br>
                ask any questions and I'll answer here and try to <br>
                update documentation. <br>
                                                          <br>
                Meanwhile I'll prepare missing bits for steps (2) and <br>
                (3). <br>
                Hopefully everything is in place for step (4), but we <br>
                need to make steps (2) and (3) first. <br>
                                                          <br>
                Andrew. <br>
                                                          <br>
                On 8/18/23 21:40, Andrew Rybchenko wrote: <br>
                                                          <blockquote
                                                          type="cite">               
                                                          Hi Adam, <br>
                                                          <br>
                > I've conferred with the rest of the team, and we <br>
                think it would be best to move forward with mainly <br>
                option B. <br>
                                                          <br>
                OK, I'll provide the sample on Monday for you. It is <br>
                almost ready right now, but I need to double-check <br>
                it before publishing. <br>
                                                          <br>
                Regards, <br>
                Andrew. <br>
                                                          <br>
                On 8/17/23 20:03, Adam Hassick wrote: <br>
                                                          <blockquote
                                                          type="cite">               
                                                          Hi Andrew, <br>
                                                          <br>
                I'm adding the CI mailing list to this <br>
                conversation. Others in the community might find <br>
                this conversation valuable. <br>
                                                          <br>
                We do want to run testing on a regular basis. The <br>
                Jenkins integration will be very useful for us, as <br>
                most of our CI is orchestrated by Jenkins. <br>
                I've conferred with the rest of the team, and we <br>
                think it would be best to move forward with mainly <br>
                option B. <br>
                If you would like to know anything about our <br>
                testbeds that would help you with creating an <br>
                example ts-rigs repo, I'd be happy to answer any <br>
                questions you have. <br>
                                                          <br>
                We have multiple test rigs (we call these <br>
                "DUT-tester pairs") that we run our existing <br>
                hardware testing on, with differing network <br>
                hardware and CPU architecture. I figured this might <br>
                be an important detail. <br>
                                                          <br>
                Thanks, <br>
                Adam <br>
                                                          <br>
                On Thu, Aug 17, 2023 at 11:44 AM Andrew Rybchenko <br>
                <<a href="mailto:andrew.rybchenko@oktetlabs.ru"
                                                          target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">andrew.rybchenko@oktetlabs.ru</a>
                                                          <br>
                <a href="mailto:andrew.rybchenko@oktetlabs.ru"
                                                          target="_blank"
moz-do-not-send="true"><mailto:andrew.rybchenko@oktetlabs.ru></a>>
                                                          wrote: <br>
                                                          <br>
                    Greatings Adam, <br>
                                                          <br>
                    I'm happy to hear that you're trying to bring <br>
                    it up. <br>
                                                          <br>
                    As I understand the final goal is to run it on <br>
                    regular basis. So, we need to make it properly <br>
                    from the very beginning. <br>
                    Bring up of all features consists of 4 steps: <br>
                                                          <br>
                    1. Create site-specific repository (we call it <br>
                    ts-rigs) which contains information about test <br>
                    rigs and other site-specific information like <br>
                    where to send mails, where to store logs etc. <br>
                    It is required for manual execution as well, <br>
                    since test rigs description is essential. I'll <br>
                    return to the topic below. <br>
                                                          <br>
                    2. Setup logs storage for automated runs. <br>
                    Basically it is a disk space plus apache2 web <br>
                    server with few CGI scripts which help a lot to <br>
                    save disk space. <br>
                                                          <br>
                    3. Setup Bublik web application which provides <br>
                    web interface to view testing results. Same as <br>
                    <a href="https://ts-factory.io/bublik"
                                                          target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://ts-factory.io/bublik</a>
                                                          <br>
                    <a href="https://ts-factory.io/bublik"
                                                          target="_blank"
moz-do-not-send="true"><https://ts-factory.io/bublik></a> <br>
                                                          <br>
                    4. Setup Jenkins to run tests on regularly, <br>
                    save logs in log storage (2) and import it to <br>
                    bublik (3). <br>
                                                          <br>
                    Last few month we spent on our homework to make <br>
                    it simpler to bring up automated execution <br>
                    using Jenkins - <br>
                    <a href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ts-factory/te-jenkins</a>
                                                          <br>
                    <a href="https://github.com/ts-factory/te-jenkins"
target="_blank" moz-do-not-send="true"><https://github.com/ts-factory/te-jenkins></a>
                                                          <br>
                    Corresponding bits in dpdk-ethdev-ts will be <br>
                    available tomorrow. <br>
                                                          <br>
                    Let's return to the step (1). <br>
                                                          <br>
                    Unfortunately there is no publicly available <br>
                    example of the ts-rigs repository since <br>
                    sensitive site-specific information is located <br>
                    there. But I'm ready to help you to create it <br>
                    for UNH. I see two options here: <br>
                                                          <br>
                    (A) I'll ask questions and based on your <br>
                    answers will create the first draft with my <br>
                    comments. <br>
                                                          <br>
                    (B) I'll make a template/example ts-rigs repo, <br>
                    publish it and you'll create UNH ts-rigs based <br>
                    on it. <br>
                                                          <br>
                    Of course, I'll help to debug and finally bring <br>
                    it up in any case. <br>
                                                          <br>
                    (A) is a bit simpler for me and you, but (B) is <br>
                    a bit more generic and will help other <br>
                    potential users to bring it up. <br>
                    We can combine (A)+(B). I.e. start from (A). <br>
                    What do you think? <br>
                                                          <br>
                    Thanks, <br>
                    Andrew. <br>
                                                          <br>
                    On 8/17/23 15:18, Konstantin Ushakov wrote: <br>
                                                          <blockquote
                                                          type="cite">                   
                                                          Greetings
                                                          Adam, <br>
                                                          <br>
                                                          <br>
                    Thanks for contacting us. I copy Andrew who <br>
                    would be happy to help <br>
                                                          <br>
                    Thanks, <br>
                    Konstantin <br>
                                                          <br>
                                                          <blockquote
                                                          type="cite">                   
                                                          On 16 Aug
                                                          2023, at
                                                          21:50, Adam
                                                          Hassick <br>
                    <a href="mailto:ahassick@iol.unh.edu"
                                                          target="_blank"
moz-do-not-send="true"><ahassick@iol.unh.edu></a> <br>
                    <a href="mailto:ahassick@iol.unh.edu"
                                                          target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a> wrote: <br>
                                                          <br>
                     <br>
                    Greetings Konstantin, <br>
                                                          <br>
                    I am in the process of setting up the DPDK <br>
                    Poll Mode Driver test suite as an addition to <br>
                    our testing coverage for DPDK at the UNH lab. <br>
                                                          <br>
                    I have some questions about how to set the <br>
                    test suite arguments. <br>
                                                          <br>
                    I have been able to configure the Test Engine <br>
                    to connect to the hosts in the testbed. The <br>
                    RCF, Configurator, and Tester all begin to <br>
                    run, however the prelude of the test suite <br>
                    fails to run. <br>
                                                          <br>
                    <a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters</a>
                                                          <a
href="https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters"
target="_blank" moz-do-not-send="true"><https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters></a>
                                                          <br>
                                                          <br>
                    The documentation mentions that there are <br>
                    several test parameters for the test suite, <br>
                    like for the IUT test link MAC, etc. These <br>
                    seem like they would need to be set somewhere <br>
                    to run many of the tests. <br>
                                                          <br>
                    I see in the Test Engine documentation, there <br>
                    are instructions on how to create new <br>
                    parameters for test suites in the Tester <br>
                    configuration, but there is nothing in the <br>
                    user guide or in the Tester guide for how to <br>
                    set the arguments for the parameters when <br>
                    running the test suite that I can find. I'm <br>
                    not sure if I need to write my own Tester <br>
                    config, or if I should be setting these in <br>
                    some other way. <br>
                                                          <br>
                    How should these values be set? <br>
                                                          <br>
                    I'm also not sure what environment <br>
                    variables/arguments are strictly necessary or <br>
                    which are optional. <br>
                                                          <br>
                    Regards, <br>
                    Adam <br>
                                                          <br>
                    --                     *Adam Hassick* <br>
                    Senior Developer <br>
                    UNH InterOperability Lab <br>
                    <a href="mailto:ahassick@iol.unh.edu"
                                                          target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
                                                          <br>
                    <a href="mailto:ahassick@iol.unh.edu"
                                                          target="_blank"
moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a> <br>
                    <a href="http://iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true">iol.unh.edu</a>
                                                          <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                    +1 (603) 475-8248 <br>
                                                          </blockquote>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <br>
                --                 *Adam Hassick* <br>
                Senior Developer <br>
                UNH InterOperability Lab <br>
                <a href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a> <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                <a href="http://iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true">iol.unh.edu</a>
                                                          <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                +1 (603) 475-8248 <br>
                                                          </blockquote>
                                                          <br>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <br>
                                                                      --
                                                                     
                                                          *Adam Hassick*
                                                          <br>
                                                                     
                                                          Senior
                                                          Developer <br>
                                                                     
                                                          UNH
                                                          InterOperability
                                                          Lab <br>
                                                                      <a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a> <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                                                                      <a
href="http://iol.unh.edu" target="_blank" moz-do-not-send="true">iol.unh.edu</a>
                                                          <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                                                                      +1
                                                          (603) 475-8248
                                                          <br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                                  --
                                                                  *Adam
                                                          Hassick* <br>
                                                                  Senior
                                                          Developer <br>
                                                                  UNH
                                                          InterOperability
                                                          Lab <br>
                                                                  <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
                                                          <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                                                                  <a
                                                          href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a> <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                                                                  +1
                                                          (603) 475-8248
                                                          <br>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <br>
                                                              --    
                                                          *Adam Hassick*
                                                          <br>
                                                              Senior
                                                          Developer <br>
                                                              UNH
                                                          InterOperability
                                                          Lab <br>
                                                              <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
                                                          <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                                                              <a
                                                          href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a> <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                                                              +1 (603)
                                                          475-8248 <br>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          -- <br>
                                                          *Adam Hassick*
                                                          <br>
                                                          Senior
                                                          Developer <br>
                                                          UNH
                                                          InterOperability
                                                          Lab <br>
                                                          <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a>
                                                          <a
                                                          href="mailto:ahassick@iol.unh.edu"
target="_blank" moz-do-not-send="true"><mailto:ahassick@iol.unh.edu></a>
                                                          <br>
                                                          <a
                                                          href="http://iol.unh.edu"
target="_blank" moz-do-not-send="true">iol.unh.edu</a> <a
                                                          href="https://www.iol.unh.edu/"
target="_blank" moz-do-not-send="true"><https://www.iol.unh.edu/></a>
                                                          <br>
                                                          +1 (603)
                                                          475-8248 <br>
                                                          </blockquote>
                                                          <br>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br
                                                          clear="all">
                                                          <br>
                                                          <span
                                                          class="gmail_signature_prefix">--
                                                          </span><br>
                                                          <div dir="ltr"
class="gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div><b><span
style="background-color:rgb(255,255,255)"><span
                                                          style="color:rgb(102,102,102)">Adam
                                                          Hassick</span></span></b><br>
                                                          </div>
                                                          <span
                                                          style="color:rgb(102,102,102)"></span></div>
                                                          <div><span
                                                          style="color:rgb(102,102,102)">Senior
                                                          Developer</span></div>
                                                          <div><span
                                                          style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
                                                          style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
                                                          style="color:rgb(102,102,102)"></span></div>
                                                          <div><span
                                                          style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                                                          </span></div>
                                                          <div><span
                                                          style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
                                                          </span></div>
                                                          +1 (603)
                                                          475-8248<br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                        </blockquote>
                                                        <br>
                                                      </div>
                                                    </blockquote>
                                                  </div>
                                                  <br clear="all">
                                                  <br>
                                                  <span
                                                    class="gmail_signature_prefix">--
                                                  </span><br>
                                                  <div dir="ltr"
                                                    class="gmail_signature">
                                                    <div dir="ltr">
                                                      <div>
                                                        <div><b><span
                                                          style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
                                                        </div>
                                                        <span
                                                          style="color:rgb(102,102,102)"></span></div>
                                                      <div><span
                                                          style="color:rgb(102,102,102)">Senior
                                                          Developer</span></div>
                                                      <div><span
                                                          style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
                                                          style="background-color:rgb(255,255,255)">UNH
InterOperability Lab</span></span></span><span
                                                          style="color:rgb(102,102,102)"></span></div>
                                                      <div><span
                                                          style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                          moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                                                        </span></div>
                                                      <div><span
                                                          style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
                                                        </span></div>
                                                      +1 (603) 475-8248<br>
                                                    </div>
                                                  </div>
                                                </blockquote>
                                                <br>
                                              </blockquote>
                                              <br>
                                            </div>
                                          </blockquote>
                                        </div>
                                        <br clear="all">
                                        <br>
                                        <span
                                          class="gmail_signature_prefix">--
                                        </span><br>
                                        <div dir="ltr"
                                          class="gmail_signature">
                                          <div dir="ltr">
                                            <div>
                                              <div><b><span
                                                    style="background-color:rgb(255,255,255)"><span
style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
                                              </div>
                                              <span
                                                style="color:rgb(102,102,102)"></span></div>
                                            <div><span
                                                style="color:rgb(102,102,102)">Senior
                                                Developer</span></div>
                                            <div><span
                                                style="color:rgb(102,102,102)"><span
style="color:rgb(11,83,148)"><span
                                                    style="background-color:rgb(255,255,255)">UNH
                                                    InterOperability Lab</span></span></span><span
style="color:rgb(102,102,102)"></span></div>
                                            <div><span
                                                style="color:rgb(102,102,102)"><a
href="mailto:ahassick@iol.unh.edu" target="_blank"
                                                  moz-do-not-send="true"
class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                                              </span></div>
                                            <div><span
                                                style="color:rgb(102,102,102)"><a
href="https://www.iol.unh.edu/" target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
                                              </span></div>
                                            +1 (603) 475-8248<br>
                                          </div>
                                        </div>
                                      </blockquote>
                                      <br>
                                    </div>
                                  </blockquote>
                                </div>
                              </div>
                            </div>
                          </blockquote>
                          <br>
                        </div>
                      </blockquote>
                    </div>
                    <br clear="all">
                    <br>
                    <span class="gmail_signature_prefix">-- </span><br>
                    <div dir="ltr" class="gmail_signature">
                      <div dir="ltr">
                        <div>
                          <div><b><span
                                style="background-color:rgb(255,255,255)"><span
                                  style="color:rgb(102,102,102)">Adam
                                  Hassick</span></span></b><br>
                          </div>
                          <span style="color:rgb(102,102,102)"></span></div>
                        <div><span style="color:rgb(102,102,102)">Senior
                            Developer</span></div>
                        <div><span style="color:rgb(102,102,102)"><span
                              style="color:rgb(11,83,148)"><span
                                style="background-color:rgb(255,255,255)">UNH
                                InterOperability Lab</span></span></span><span
                            style="color:rgb(102,102,102)"></span></div>
                        <div><span style="color:rgb(102,102,102)"><a
                              href="mailto:ahassick@iol.unh.edu"
                              target="_blank" moz-do-not-send="true"
                              class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                          </span></div>
                        <div><span style="color:rgb(102,102,102)"><a
                              href="https://www.iol.unh.edu/"
                              target="_blank" moz-do-not-send="true">iol.unh.edu</a><br>
                          </span></div>
                        +1 (603) 475-8248<br>
                      </div>
                    </div>
                  </blockquote>
                  <br>
                </div>
              </blockquote>
            </div>
            <br clear="all">
            <br>
            <span class="gmail_signature_prefix">-- </span><br>
            <div dir="ltr" class="gmail_signature">
              <div dir="ltr">
                <div>
                  <div><b><span
                        style="background-color:rgb(255,255,255)"><span
                          style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
                  </div>
                  <span style="color:rgb(102,102,102)"></span></div>
                <div><span style="color:rgb(102,102,102)">Senior
                    Developer</span></div>
                <div><span style="color:rgb(102,102,102)"><span
                      style="color:rgb(11,83,148)"><span
                        style="background-color:rgb(255,255,255)">UNH
                        InterOperability Lab</span></span></span><span
                    style="color:rgb(102,102,102)"></span></div>
                <div><span style="color:rgb(102,102,102)"><a
                      href="mailto:ahassick@iol.unh.edu" target="_blank"
                      moz-do-not-send="true"
                      class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
                  </span></div>
                <div><span style="color:rgb(102,102,102)"><a
                      href="https://www.iol.unh.edu/" target="_blank"
                      moz-do-not-send="true">iol.unh.edu</a><br>
                  </span></div>
                +1 (603) 475-8248<br>
              </div>
            </div>
          </blockquote>
        </div>
        <br clear="all">
        <br>
        <span class="gmail_signature_prefix">-- </span><br>
        <div dir="ltr" class="gmail_signature">
          <div dir="ltr">
            <div>
              <div><b><span style="background-color:rgb(255,255,255)"><span
                      style="color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
              </div>
              <span style="color:rgb(102,102,102)"></span></div>
            <div><span style="color:rgb(102,102,102)">Senior Developer</span></div>
            <div><span style="color:rgb(102,102,102)"><span
                  style="color:rgb(11,83,148)"><span
                    style="background-color:rgb(255,255,255)">UNH
                    InterOperability Lab</span></span></span><span
                style="color:rgb(102,102,102)"></span></div>
            <div><span style="color:rgb(102,102,102)"><a
                  href="mailto:ahassick@iol.unh.edu" target="_blank"
                  moz-do-not-send="true" class="moz-txt-link-freetext">ahassick@iol.unh.edu</a><br>
              </span></div>
            <div><span style="color:rgb(102,102,102)"><a
                  href="https://www.iol.unh.edu/" target="_blank"
                  moz-do-not-send="true">iol.unh.edu</a><br>
              </span></div>
            +1 (603) 475-8248<br>
          </div>
        </div>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </body>
</html>