Trying to get the 'make check' output out of dockerland.

[ndp-opendap]

Description

This is ready for review. With this PR, our build-docker jobs in BES TravisCI (.travis.yml), the Docker file runs make check (and not make check -k. Should we switch?) and captures the status without failing immediately when the tests fail. The status is written to a file and copied out of the docker image along with the test log file inventory and the test logs tarball. This way the calling code in build-rhel-docker.sh and ultimately .travis.yml can upload the test logs to our S3 bucket and only then fail the build.

          ./travis/build-rhel-docker.sh
          build_status=$?
          echo "# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -" >&2
          echo "# (In Travis) build_status: $build_status" >&2
          ./travis/upload-test-results.sh
          echo "# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -" >&2
          
          if [ "$build_status" -ne 0 ]; then
            echo "# ERROR - Docker build failed, exiting. build_status: $build_status" >&2
            exit $build_status;
          fi

Reference ticket: HYRAX-####

TODO

Tasks

• Ticket exists and is linked in title
• Tests added/updated
• Dead code removed
• No TODOs added

[jgallagher59701]

I think there is a problem where the build for EL8 can pass but tests fail. I'll put that in a comment because it's a long comment and it's not something I can isolate to specific piece of text.

[jgallagher59701]

Here in the EL8 build output in Travis, I see:

# upload-test-result() - Pushing /tmp/bes-autotest-7686.4-logs.tgz to S3
upload: ../../../../../tmp/bes-autotest-7686.4-logs.tgz to s3://opendap.travis.tests/bes-autotest-7686.4-logs.tgz
# upload-test-result() - END
# upload-test-result() - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The command "./travis/build-rhel-docker.sh
build_status=$?
echo "# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -" >&2
echo "# (In Travis) build_status: $build_status" >&2
./travis/upload-test-results.sh
echo "# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -" >&2
if [ "$build_status" -ne 0 ]; then
  echo "# ERROR - Docker build failed, exiting. build_status: $build_status" >&2
  exit $build_status;
fi
" exited with 0.
0.38s$ export BES_VERSION="$(docker run --rm ${SNAPSHOT_IMAGE_TAG} -c 'cat bes_VERSION')"
export BUILD_VERSION_TAG="opendap/${DOCKER_NAME}:${BES_VERSION}-${DIST}"
echo "# Tagging image with BES version: ${BUILD_VERSION_TAG}"
docker tag ${SNAPSHOT_IMAGE_TAG} ${BUILD_VERSION_TAG}
# Tagging image with BES version: opendap/bes_core:3.21.1-1862-el8
The command "export BES_VERSION="$(docker run --rm ${SNAPSHOT_IMAGE_TAG} -c 'cat bes_VERSION')"
export BUILD_VERSION_TAG="opendap/${DOCKER_NAME}:${BES_VERSION}-${DIST}"
echo "# Tagging image with BES version: ${BUILD_VERSION_TAG}"
docker tag ${SNAPSHOT_IMAGE_TAG} ${BUILD_VERSION_TAG}
" exited with 0.
cache.2
store build cache
Skipping a deployment with the s3 provider because the current build is a pull request.
Skipping a deployment with the script provider because the current build is a pull request.
Done. Your build exited with 0.

Which says that that there was no error.

But, when I looked at s3://opendap.travis.tests/bes-autotest-7686.4-logs.tgz and I looked around, I found in bes/dap/tests that two tests (05 and 06) failed (Segmentation fault). Here's the testsuite.log for test 05:

(base) ➜  tests git:(master) ✗ more testsuite.dir/05/testsuite.log 
#                             -*- compilation -*-
5. testsuite.at:13: testing BESCMD $abs_srcdir/bescmd/function_result_unwrap_dap.bescmd ...
./testsuite.at:13: besstandalone -c $abs_builddir/$bes_conf -i $input
--- /dev/null   2026-07-13 12:14:48.308007132 +0000
+++ /home/bes_user/bes/dap/tests/testsuite.dir/at-groups/5/stderr       2026-07-13 12:15:14.792007386 +0000
@@ -0,0 +1,4 @@
+Exception: The DMR was not well formed. At line 74: Extra content at the end of the document
+
+At line 74: The document contained unbalanced tags.
+/home/bes_user/bes/dap/tests/testsuite.dir/at-groups/5/test-source: line 89:  3046 Segmentation fault      (core dumped) besstandalone -c $abs_builddir/$bes_conf -i $input
stdout:
./testsuite.at:13: exit code was 139, expected 0
5. testsuite.at:13: 5. BESCMD $abs_srcdir/bescmd/function_result_unwrap_dap.bescmd (testsuite.at:13): expected failure (testsuite.at:13)

Searching, I found:

#32 26.32 ## ----------------------------------------------- ##
#32 26.32 ## bes 3.21.1 test suite: bes/dap/tests testsuite. ##
#32 26.32 ## ----------------------------------------------- ##
#32 26.39   1: BESCMD $abs_srcdir/bescmd/agg_array_dds_one_ce.bescmd ok
#32 26.43   2: BESCMD $abs_srcdir/bescmd/agg_array_dods_one_ce.bescmd ok
#32 26.48   3: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dds.bescmd ok
#32 26.52   4: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dods.bescmd ok
#32 26.56   5: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dap.bescmd expected failure (testsuite.at:13)
#32 26.60   6: BESCMD $abs_srcdir/bescmd/dap4_ce_function-01.bescmd expected failure (testsuite.at:15)
#32 26.64   7: BESCMD $abs_srcdir/bescmd/fnoc.das.bescmd       ok
#32 26.68   8: BESCMD $abs_srcdir/bescmd/fnoc.ddx.bescmd       ok
#32 26.72   9: BESCMD $abs_srcdir/bescmd/fnoc.dmr.bescmd       ok
#32 26.77  10: BESCMD $abs_srcdir/bescmd/dap4_response_too_big.bescmd ok
#32 26.83  11: BESCMD $abs_srcdir/bescmd/dap4_variable_too_big.bescmd ok
#32 26.90  12: BESCMD $abs_srcdir/bescmd/dap4_var_and_response_too_big.bescmd ok

I think that there are places in the Travis script when $? is used and that can be tricky because Travis uses that for its own purpose (I've read that, I don't know much more about it). That might be the problem.

[ndp-opendap]

Interesting. Both tests 5 and 6 are expected failures:

#32 26.56   5: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dap.bescmd expected failure (testsuite.at:13)
#32 26.60   6: BESCMD $abs_srcdir/bescmd/dap4_ce_function-01.bescmd expected failure (testsuite.at:15)

so I think that's maybe why there were no errors reported??

[jgallagher59701]

This could be part of the issue I talk about in the PR comment. Here's a link to the Travis CI docs: https://docs.travis-ci.com/user/job-lifecycle/#use-the--command Bottom line, don't use $? in the script part of the Travis CI yaml.

[ndp-opendap]

But that's not "in" .travis.yml

[jgallagher59701]

Interesting. Both tests 5 and 6 are expected failures:

#32 26.56   5: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dap.bescmd expected failure (testsuite.at:13)
#32 26.60   6: BESCMD $abs_srcdir/bescmd/dap4_ce_function-01.bescmd expected failure (testsuite.at:15)

so I think that's maybe why there were no errors reported??

Yeah, my mistake

[ndp-opendap]

Ok @jgallagher59701 - I addressed the $? + .travis.yml issue.I think it's working correctly, but the only test failures I am seeing in the logs are expected failures.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[kneumiller]

I believe this is ready to be merged. I'll write up the ticket for the sonarscan/hdf failures.