Skip to content

Fix sdkdev integration failures: gate REGION_DURABLE, cover DURABLE, pin OSS image - #176

Merged
chrishagglund-ship-it merged 3 commits into
mainfrom
fix/skip-region-durable-tests-and-improve-oss-version-pin
Sep 10, 2026
Merged

Fix sdkdev integration failures: gate REGION_DURABLE, cover DURABLE, pin OSS image#176
chrishagglund-ship-it merged 3 commits into
mainfrom
fix/skip-region-durable-tests-and-improve-oss-version-pin

Conversation

@chrishagglund-ship-it

@chrishagglund-ship-it chrishagglund-ship-it commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

What's failing

Four tests in WorkflowExecutor.test.ts have been red on integration v5 sdkdev
(shard 1/3, all of Node 20/22/24):

● Should execute complex workflow with DURABLE + TARGET_WORKFLOW
  [Conductor SDK Error]: Failed to execute workflow:
  REGION_DURABLE consistency requested but region replication is not
  enabled/configured on this node

Note the mismatch: the test says DURABLE, the server says REGION_DURABLE.

Why

The cases were named DURABLE + * but had always passed
Consistency.REGION_DURABLE — the section comment even said
// REGION_DURABLE consistency tests. Nothing in this repo changed. They went red
when sdkdev picked up a server build carrying the active/active work
(ea0a37ad74, "region-durable
workflow start (synchronous cross-region push, fail-closed)"
).

Before that change, REGION_DURABLE was an inert enum value that fell through the
same scheduleWorkflow branch as DURABLE, so the flag was accepted and silently
ignored. After it, a node with no replicator wired rejects the start outright rather
than appearing to honour a guarantee it cannot provide.

The server is right. sdkdev is a single-node instance that is not meant to have
replication. The tests were green under false pretences: they asked for cross-region
durability, got a plain local start, and asserted success.

java-sdk hit this identically and fixed it in
java-sdk#167, which dates
the sdkdev upgrade precisely — 2026-08-11, orkes-conductor 5.3.0 → 5.5.0.

Changes

1. Gate the REGION_DURABLE cases — renamed to REGION_DURABLE + * to match what
they actually request, gated behind CONDUCTOR_REGION_DURABLE_ENABLED via a new
testForRegionDurable helper.

Explicit opt-in rather than catch-the-500-and-skip, following java-sdk's reasoning:
auto-skip would restore exactly the silent green that hid this for months, and would
mask a genuine replication regression on a cluster where it is meant to work. Same
env var name as java-sdk so both SDKs configure identically.

2. Add four real DURABLE + * cases. Consistency.DURABLE appeared nowhere
in the suite despite being the default level, and it takes a different executor path
than SYNCHRONOUS:

Consistency Executor path
SYNCHRONOUS decideWithLock + pushToDeciderQueueWithTimeout, inline
DURABLE scheduleWorkflow — async decider queue

Less new coverage than restored coverage — it is what the REGION_DURABLE cases were
really exercising all along, back when the server ignored the flag.

3. Write the OSS Conductor image tag in exactly one place. Previously the OSS job
tracked latest, so an upstream release could turn it red on an unrelated PR — the
same class of surprise as the sdkdev upgrade above. The tag now resolves like this:

Context Tag comes from
CI, normal run E2E_TEST_OSS_CONDUCTOR_VERSION org variable, or the workflow_dispatch input
CI, fork PR the image: default in scripts/docker-compose-oss.yaml — GitHub withholds vars.* from forks exactly as it withholds secrets
Local the same compose default, unless run-integration-oss.sh --version <tag> overrides it
CI, non-fork run with no org variable nothing — the preflight step fails with an actionable message

The default itself is written once, on the image: line of the compose file
(3.32.3). run-integration-oss.sh applies no default of its own and exports
OSS_CONDUCTOR_VERSION only when --version actually supplied one, so a plain local
run and a fork CI run reach the identical image by the identical path — which is what
lets the script reproduce a CI failure. This removed a hardcoded fallback rather than
adding a mechanism.

4. Pull the image in CI before starting the stack. Redundant on a GitHub-hosted
runner, whose VM is ephemeral and starts with no cached copy, so up pulls anyway.
Kept regardless: it costs no extra network pull (up then finds the image locally),
it splits "couldn't pull the image" from "the stack didn't come up" into two distinct
red steps, and it is what stops a mutable tag going stale if this job ever moves to a
self-hosted runner with a warm Docker daemon — the same reason the local script pulls.
It also logs the tag actually in use, which now matters because a fork PR's tag is not
spelled out in the workflow.

Test matrix after this change

Cases Consistency Gated
SYNC + * ×4 SYNCHRONOUS no (unchanged)
DURABLE + * ×4 DURABLE no — new
REGION_DURABLE + * ×4 REGION_DURABLE yes

sdkdev goes from 4 failures → 4 skips + 4 new passing cases.

Review feedback

All three comments are addressed in 5227051:

Comment Resolution
CI pins the image tag but a local run defaults to latest, so the script won't reproduce a CI failure Change 3 — one default, written in the compose file, reached by both paths
The script pulls the image before starting the stack, but CI doesn't Change 4 — new Pull Conductor OSS image step
agent-e2e.yml already calls this CONDUCTOR_OSS_VERSION — use one name, or say why they're separate Comment added at the definition in agent-e2e.yml. They are separate on purpose: that one is a Maven Central artifact version for the conductor-server boot JAR, this one a Docker Hub image tag. The two registries publish different version sets — Maven has no latest — and the agent suites carry a floor this job does not (≥ 3.32.0-rc.8, for the /agent/* control plane and TaskDef.runtimeMetadata persistence). Converging them is possible once the org variable holds a concrete version, but that is a rename plus a bump of a stale pin, not this PR.

Notes for reviewers

  • No CI configuration needs CONDUCTOR_REGION_DURABLE_ENABLED. Gated-off is the
    correct state for a single-region server; java-sdk likewise sets it in no workflow.
    Set it only against a target with cross-region replication configured.
  • E2E_TEST_OSS_CONDUCTOR_VERSION is currently set to latest at the org level.
    So the pin in change 3 is nominal for normal CI runs until someone sets that variable
    to a real version — today it is the compose default that actually holds the line, on
    fork PRs and locally. The variable must also have this repo in its repository access
    policy, or a non-fork run fails the preflight step by design.
  • Known, pre-existing, untouched: on a waitForSeconds timeout the execute endpoint
    returns toWorkflowRun(...), which never sets responseType — so a timeout surfaces
    as a confusing undefined mismatch rather than "it timed out." Affects all 12 combos
    equally and predates this PR; worth a separate fix.

@chrishagglund-ship-it
chrishagglund-ship-it marked this pull request as ready for review September 9, 2026 19:27

@ambiorix2099 ambiorix2099 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Approved with comments.

Comment thread .github/workflows/pull_request.yml Outdated
Comment thread scripts/run-integration-oss.sh Outdated
Comment thread .github/workflows/pull_request.yml
chrishagglund-ship-it added a commit that referenced this pull request Sep 10, 2026
Review feedback on #176. Three separate points, all about the Conductor OSS
image tag having more than one home.

One tag, one home. The tag was written twice: FORK_PR_FALLBACK_VERSION in the
integration-tests-oss job, and `latest` as the script's default. A local run
therefore could not reproduce a CI failure, which is the whole point of the
script existing. Rather than teach one of them to read the other, make the
`image:` line of docker-compose-oss.yaml the only place it is written and let
both fall through to it: the script no longer applies a default at all and
exports OSS_CONDUCTOR_VERSION only when --version actually supplied one, and
the fork-PR branch of the resolve step now leaves the variable unset instead of
pinning its own copy. Fork CI and a plain local run land on the identical
image by taking the identical path, with no YAML parsing on either side and one
hardcode removed rather than a mechanism added. The non-fork empty case still
fails loudly. The script's "Using ..." and "Pulling ..." lines now come from
`compose config --images conductor-server` instead of reconstructing the tag,
so they stay honest whether it came from the flag or the compose default.

Worth stating plainly: E2E_TEST_OSS_CONDUCTOR_VERSION is currently set to
`latest` at the org level, so the pin this series added is nominal until
someone sets it to a real version. The compose default is what actually holds
the line today, on fork PRs and locally.

Pull in CI. The script pulls before `up`, CI did not. On a GitHub-hosted
runner the VM is ephemeral and starts with no cached copy of this image, so
`up` pulls anyway and the step is redundant today — kept regardless, because it
costs no extra network pull (`up` then finds the image locally), it splits
"couldn't pull the image" from "the stack didn't come up" into two distinct red
steps, and it is what stops a mutable tag going stale the day this job moves to
a self-hosted runner with a warm Docker daemon. It also prints the tag in use,
which matters now that a fork PR's tag is not spelled out in the workflow.

agent-e2e's near-namesake. CONDUCTOR_OSS_VERSION there reads like it should be
the same knob and is not, so say why at the definition rather than leave the
next reader to work it out. That one is a Maven Central artifact version for
the conductor-server boot JAR; this one is a Docker Hub image tag. Different
registries publish different version sets — Maven has no `latest`, which is
exactly what the org variable holds — and the agent suites additionally need
>= 3.32.0-rc.8 for the /agent/* control plane and TaskDef.runtimeMetadata
persistence they exercise. Pinned independently, deliberately. Converging the
two is possible once the org variable holds a concrete version (3.32.3 and
3.33.0-rc1 are published to both registries), but that is a bump of a stale
rc18 and a rename, not this change.

This does move the OSS harness off byte-identical with the other six SDKs,
which the previous commit had just restored. Deliberate and temporary: the
same edit applies verbatim to java-sdk and python-sdk, held back for now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…image

The four "DURABLE + *" cases in WorkflowExecutor.test.ts have been failing on
sdkdev across Node 20/22/24 (shard 1/3) with

  REGION_DURABLE consistency requested but region replication is not
  enabled/configured on this node

Nothing changed in this repo: the cases were named "DURABLE + *" but had always
passed Consistency.REGION_DURABLE. They went red when sdkdev picked up a server
build carrying the active/active work (orkes-conductor ea0a37ad74, "region-durable
workflow start"). Before it, REGION_DURABLE was an inert enum value that fell
through the same scheduleWorkflow branch as DURABLE, so the flag was accepted and
silently ignored; after it, a node with no replicator wired rejects the start
outright rather than appearing to honour a guarantee it cannot provide.

The server is right — sdkdev is a single-node instance not meant to have
replication — and the tests were green under false pretences: they asked for
cross-region durability, got a plain local start, and asserted success.

So:

- Rename the four cases to "REGION_DURABLE + *", matching what they request, and
  gate them on CONDUCTOR_REGION_DURABLE_ENABLED via a new testForRegionDurable
  helper. Explicit opt-in rather than catching the 500 and skipping: auto-skip
  would restore exactly the silent green that hid this for months, and would mask
  a genuine replication regression on a cluster where it is meant to work. Same
  gate name as the java-sdk fix (orkes-io/java-sdk#167) so both SDKs configure
  identically.

- Add four real "DURABLE + *" cases. Consistency.DURABLE appeared nowhere in the
  suite despite being the default level, and it takes a different executor path
  than SYNCHRONOUS (async decider queue via scheduleWorkflow, not an inline
  decide), so the return strategies are worth covering against it. This is not
  new coverage so much as restored: it is what the REGION_DURABLE cases were
  really exercising before the server started honouring the flag.

Also pins the OSS Conductor image instead of tracking `latest`, so an upstream
release cannot turn the OSS job red on an unrelated PR — the same class of
surprise as the sdkdev upgrade above. The tag comes from the
E2E_TEST_OSS_CONDUCTOR_VERSION org variable or a workflow_dispatch input, with a
preflight step that fails with an actionable message when neither resolves, and
scripts/run-integration-oss.sh grows a --version flag (defaulting to `latest`)
for local runs.

Net effect on CI: sdkdev goes from 4 failures to 4 skips plus 4 new passing
cases. No CI configuration needs the new variable — gated-off is correct for a
single-region server, and java-sdk likewise sets it in no workflow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three gaps against the OSS harness the other six SDKs share (ruby, python, go,
csharp, java, rust), which this repo's variant was derived from without carrying
across. Each change below is byte-identical to those, so the seven stay
diffable.

Fork PRs. GitHub withholds org/repo variables from pull_request runs on forks
exactly as it withholds secrets, so vars.E2E_TEST_OSS_CONDUCTOR_VERSION is
always "" for an outside contributor (observed in csharp-sdk#178). Resolving in
the job `env` and null-checking therefore failed every fork PR with a config
error the contributor has no power to fix — on a job that needs no secrets at
all, only a tag, since it runs a local docker-compose stack against localhost.

Resolve in a step writing $GITHUB_ENV instead, so the two ways the tag can come
back empty get different treatment: a fork PR pins FORK_PR_FALLBACK_VERSION and
keeps running, anything else still fails loudly rather than silently drifting
onto the pin. The pin is 3.32.3, matching all six.

Server logs on failure. cleanup() ran `compose down -v` without dumping
anything, and the only `compose logs` call sat inside the health-wait loop — so
logs were captured when the server never came up, but lost whenever the server
started fine and the tests then failed. `-v` drops the volumes too, so there
was no recovering them afterwards. Local runs were thus strictly worse than CI,
which already dumps via an `if: failure()` step.

Dump from the EXIT trap on any nonzero status, before the teardown and before
the --keep-up early return. `local status=$?` is first so it reflects the
failing command; `set -euo pipefail` (already present) carries a failing jest
run through the tee pipeline so the trap sees it. The health-loop dump is
removed as now redundant — its `exit 1` reaches the same trap — which also
brings this script to zero in-loop dumps like the others.

Image freshness. `docker compose up` only pulls when the image is missing
locally, so a cached mutable tag — `latest`, or a re-pushed rc — was silently
reused and nothing reported which image actually ran. Pinning the tag does not
help if a stale layer is already cached under that name. Pull conductor-server
unconditionally and echo the resolved tag first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review feedback on #176. Three separate points, all about the Conductor OSS
image tag having more than one home.

One tag, one home. The tag was written twice: FORK_PR_FALLBACK_VERSION in the
integration-tests-oss job, and `latest` as the script's default. A local run
therefore could not reproduce a CI failure, which is the whole point of the
script existing. Rather than teach one of them to read the other, make the
`image:` line of docker-compose-oss.yaml the only place it is written and let
both fall through to it: the script no longer applies a default at all and
exports OSS_CONDUCTOR_VERSION only when --version actually supplied one, and
the fork-PR branch of the resolve step now leaves the variable unset instead of
pinning its own copy. Fork CI and a plain local run land on the identical
image by taking the identical path, with no YAML parsing on either side and one
hardcode removed rather than a mechanism added. The non-fork empty case still
fails loudly. The script's "Using ..." and "Pulling ..." lines now come from
`compose config --images conductor-server` instead of reconstructing the tag,
so they stay honest whether it came from the flag or the compose default.

Worth stating plainly: E2E_TEST_OSS_CONDUCTOR_VERSION is currently set to
`latest` at the org level, so the pin this series added is nominal until
someone sets it to a real version. The compose default is what actually holds
the line today, on fork PRs and locally.

Pull in CI. The script pulls before `up`, CI did not. On a GitHub-hosted
runner the VM is ephemeral and starts with no cached copy of this image, so
`up` pulls anyway and the step is redundant today — kept regardless, because it
costs no extra network pull (`up` then finds the image locally), it splits
"couldn't pull the image" from "the stack didn't come up" into two distinct red
steps, and it is what stops a mutable tag going stale the day this job moves to
a self-hosted runner with a warm Docker daemon. It also prints the tag in use,
which matters now that a fork PR's tag is not spelled out in the workflow.

agent-e2e's near-namesake. CONDUCTOR_OSS_VERSION there reads like it should be
the same knob and is not, so say why at the definition rather than leave the
next reader to work it out. That one is a Maven Central artifact version for
the conductor-server boot JAR; this one is a Docker Hub image tag. Different
registries publish different version sets — Maven has no `latest`, which is
exactly what the org variable holds — and the agent suites additionally need
>= 3.32.0-rc.8 for the /agent/* control plane and TaskDef.runtimeMetadata
persistence they exercise. Pinned independently, deliberately. Converging the
two is possible once the org variable holds a concrete version (3.32.3 and
3.33.0-rc1 are published to both registries), but that is a bump of a stale
rc18 and a rename, not this change.

This does move the OSS harness off byte-identical with the other six SDKs,
which the previous commit had just restored. Deliberate and temporary: the
same edit applies verbatim to java-sdk and python-sdk, held back for now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@chrishagglund-ship-it
chrishagglund-ship-it force-pushed the fix/skip-region-durable-tests-and-improve-oss-version-pin branch from 02fd109 to 5227051 Compare September 10, 2026 17:31
@chrishagglund-ship-it
chrishagglund-ship-it merged commit d854d66 into main Sep 10, 2026
52 of 53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants