From 27658256ee3f77e6a588af7c3ddaee6a0beee058 Mon Sep 17 00:00:00 2001 From: mbloechli Date: Mon, 24 Aug 2026 19:58:42 +0200 Subject: [PATCH 1/2] feat(ci): build and test against the deployed RMW CI inherited whatever RMW each distro defaults to (Fast DDS), while Duatic deploys Cyclone DDS. Every green build therefore certified a middleware nobody ships. Pin rmw_cyclonedds_cpp across the matrix so the tested configuration matches the deployed one. The gap surfaced as a flake: the kilted leg of duatic_duarover failed 10 of 20 identical nightly runs on the same main commit, always in the same Nav2 sim test, always burning the full waypoint timeout after a service or action response was dropped. Only the failing runs logged Fast DDS's "failed to send response ... rmw_response.cpp", and only kilted, which is on the Fast DDS 3.x line, was affected. Jazzy, on 2.x, was 6/6 green. Two things have to line up, since naming an RMW that is not installed fails at node startup rather than falling back. The deb is installed via ADDITIONAL_DEBS as ros--rmw-cyclonedds-cpp, and the selection rides in on DOCKER_RUN_OPTS: industrial_ci builds inside Docker and only forwards the variables in its docker.env, which does not list RMW_IMPLEMENTATION, so a job-level env would never reach the tests. Both are hardcoded side by side in the industrial_ci step rather than exposed as an input, since no repo needs a different middleware. --- .github/workflows/reusable_ici.yml | 6 ++++++ README.md | 27 +++++++++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/.github/workflows/reusable_ici.yml b/.github/workflows/reusable_ici.yml index df7a868..75bc503 100644 --- a/.github/workflows/reusable_ici.yml +++ b/.github/workflows/reusable_ici.yml @@ -90,6 +90,12 @@ jobs: UPSTREAM_WORKSPACE: ${{ steps.detect.outputs.upstream_workspace }} ROS_DISTRO: ${{ inputs.ros_distro }} ROS_REPO: ${{ inputs.ros_repo }} + # industrial_ci does not forward RMW_IMPLEMENTATION into the build container + # (it is absent from its docker.env), so the selection rides in on DOCKER_RUN_OPTS. + # The matching deb goes through ADDITIONAL_DEBS, which is installed right after + # the ROS apt source is set up and fails the job if the package does not exist. + ADDITIONAL_DEBS: ros-${{ inputs.ros_distro }}-rmw-cyclonedds-cpp + DOCKER_RUN_OPTS: -e RMW_IMPLEMENTATION=rmw_cyclonedds_cpp BEFORE_INSTALL_UPSTREAM_DEPENDENCIES: ${{ steps.detect.outputs.before_install }} AFTER_INSTALL_TARGET_DEPENDENCIES: ${{ steps.detect.outputs.pip_install }} BEFORE_INIT: ${{ secrets.CI_PAT != '' && format('apt-get update -qq && apt-get install -y -qq git && git config --global url."https://{0}@github.com/".insteadOf "https://github.com/"', secrets.CI_PAT) || '' }} diff --git a/README.md b/README.md index 69652de..5c22876 100644 --- a/README.md +++ b/README.md @@ -92,6 +92,33 @@ jobs: secrets: inherit ``` +## Middleware (RMW) + +CI builds and tests on **`rmw_cyclonedds_cpp`**, the middleware Duatic deploys. Before this was +pinned, every job silently inherited whatever RMW its distro defaults to (Fast DDS), so CI was +green against a middleware nobody ships. Pinning it closes that gap: a passing build now says +something about the configuration that actually runs on a robot. + +The gap surfaced as a flake rather than as an obvious hole. The kilted leg of `duatic_duarover` +failed 10 of 20 identical nightly runs on the same `main` commit, always in the same Nav2 sim +test, always burning the full waypoint timeout because a service or action response was dropped +on the way back. Only failing runs logged `failed to send response ... rmw_response.cpp`, a Fast +DDS message, and only kilted (Fast DDS 3.x) was affected. Jazzy, on the 2.x line, was 6/6 green. + +Two pieces have to line up, because naming an RMW that is not installed is a hard failure at node +startup rather than a fallback: + +- **Installed** via industrial_ci's `ADDITIONAL_DEBS` as `ros--rmw-cyclonedds-cpp`, which + is applied right after the ROS apt source is configured and fails the job if the package is + missing. +- **Selected** via `RMW_IMPLEMENTATION` inside the container. A job-level `env:` is not enough: + industrial_ci runs the build in Docker and only forwards the variables listed in its + `docker.env`, which does not include `RMW_IMPLEMENTATION`. It is passed through + `DOCKER_RUN_OPTS: -e RMW_IMPLEMENTATION=...` instead. + +Both are hardcoded side by side in the `industrial_ci` step of `reusable_ici.yml`, so every repo +gets the same middleware and the two cannot drift apart. + ## Status badges for private repos (opt-in, gist-backed) GitHub's native workflow `badge.svg` reports at the workflow-**file** level, so it can't show From 6a2283dcc3d3ac96b9935411b40c6edb5ecc4025 Mon Sep 17 00:00:00 2001 From: mbloechli Date: Tue, 25 Aug 2026 15:48:08 +0200 Subject: [PATCH 2/2] doc: less verbose comments --- .github/workflows/reusable_ici.yml | 5 +---- README.md | 27 --------------------------- 2 files changed, 1 insertion(+), 31 deletions(-) diff --git a/.github/workflows/reusable_ici.yml b/.github/workflows/reusable_ici.yml index 75bc503..0fa28b5 100644 --- a/.github/workflows/reusable_ici.yml +++ b/.github/workflows/reusable_ici.yml @@ -90,10 +90,7 @@ jobs: UPSTREAM_WORKSPACE: ${{ steps.detect.outputs.upstream_workspace }} ROS_DISTRO: ${{ inputs.ros_distro }} ROS_REPO: ${{ inputs.ros_repo }} - # industrial_ci does not forward RMW_IMPLEMENTATION into the build container - # (it is absent from its docker.env), so the selection rides in on DOCKER_RUN_OPTS. - # The matching deb goes through ADDITIONAL_DEBS, which is installed right after - # the ROS apt source is set up and fails the job if the package does not exist. + # Install CycloneDDS as the RMW implementation ADDITIONAL_DEBS: ros-${{ inputs.ros_distro }}-rmw-cyclonedds-cpp DOCKER_RUN_OPTS: -e RMW_IMPLEMENTATION=rmw_cyclonedds_cpp BEFORE_INSTALL_UPSTREAM_DEPENDENCIES: ${{ steps.detect.outputs.before_install }} diff --git a/README.md b/README.md index 5c22876..69652de 100644 --- a/README.md +++ b/README.md @@ -92,33 +92,6 @@ jobs: secrets: inherit ``` -## Middleware (RMW) - -CI builds and tests on **`rmw_cyclonedds_cpp`**, the middleware Duatic deploys. Before this was -pinned, every job silently inherited whatever RMW its distro defaults to (Fast DDS), so CI was -green against a middleware nobody ships. Pinning it closes that gap: a passing build now says -something about the configuration that actually runs on a robot. - -The gap surfaced as a flake rather than as an obvious hole. The kilted leg of `duatic_duarover` -failed 10 of 20 identical nightly runs on the same `main` commit, always in the same Nav2 sim -test, always burning the full waypoint timeout because a service or action response was dropped -on the way back. Only failing runs logged `failed to send response ... rmw_response.cpp`, a Fast -DDS message, and only kilted (Fast DDS 3.x) was affected. Jazzy, on the 2.x line, was 6/6 green. - -Two pieces have to line up, because naming an RMW that is not installed is a hard failure at node -startup rather than a fallback: - -- **Installed** via industrial_ci's `ADDITIONAL_DEBS` as `ros--rmw-cyclonedds-cpp`, which - is applied right after the ROS apt source is configured and fails the job if the package is - missing. -- **Selected** via `RMW_IMPLEMENTATION` inside the container. A job-level `env:` is not enough: - industrial_ci runs the build in Docker and only forwards the variables listed in its - `docker.env`, which does not include `RMW_IMPLEMENTATION`. It is passed through - `DOCKER_RUN_OPTS: -e RMW_IMPLEMENTATION=...` instead. - -Both are hardcoded side by side in the `industrial_ci` step of `reusable_ici.yml`, so every repo -gets the same middleware and the two cannot drift apart. - ## Status badges for private repos (opt-in, gist-backed) GitHub's native workflow `badge.svg` reports at the workflow-**file** level, so it can't show