Fix sdkdev integration failures: gate REGION_DURABLE, cover DURABLE, pin OSS image - #176
Merged
chrishagglund-ship-it merged 3 commits intoSep 10, 2026
Conversation
chrishagglund-ship-it
marked this pull request as ready for review
September 9, 2026 19:27
ambiorix2099
approved these changes
Sep 9, 2026
chrishagglund-ship-it
added a commit
that referenced
this pull request
Sep 10, 2026
Review feedback on #176. Three separate points, all about the Conductor OSS image tag having more than one home. One tag, one home. The tag was written twice: FORK_PR_FALLBACK_VERSION in the integration-tests-oss job, and `latest` as the script's default. A local run therefore could not reproduce a CI failure, which is the whole point of the script existing. Rather than teach one of them to read the other, make the `image:` line of docker-compose-oss.yaml the only place it is written and let both fall through to it: the script no longer applies a default at all and exports OSS_CONDUCTOR_VERSION only when --version actually supplied one, and the fork-PR branch of the resolve step now leaves the variable unset instead of pinning its own copy. Fork CI and a plain local run land on the identical image by taking the identical path, with no YAML parsing on either side and one hardcode removed rather than a mechanism added. The non-fork empty case still fails loudly. The script's "Using ..." and "Pulling ..." lines now come from `compose config --images conductor-server` instead of reconstructing the tag, so they stay honest whether it came from the flag or the compose default. Worth stating plainly: E2E_TEST_OSS_CONDUCTOR_VERSION is currently set to `latest` at the org level, so the pin this series added is nominal until someone sets it to a real version. The compose default is what actually holds the line today, on fork PRs and locally. Pull in CI. The script pulls before `up`, CI did not. On a GitHub-hosted runner the VM is ephemeral and starts with no cached copy of this image, so `up` pulls anyway and the step is redundant today — kept regardless, because it costs no extra network pull (`up` then finds the image locally), it splits "couldn't pull the image" from "the stack didn't come up" into two distinct red steps, and it is what stops a mutable tag going stale the day this job moves to a self-hosted runner with a warm Docker daemon. It also prints the tag in use, which matters now that a fork PR's tag is not spelled out in the workflow. agent-e2e's near-namesake. CONDUCTOR_OSS_VERSION there reads like it should be the same knob and is not, so say why at the definition rather than leave the next reader to work it out. That one is a Maven Central artifact version for the conductor-server boot JAR; this one is a Docker Hub image tag. Different registries publish different version sets — Maven has no `latest`, which is exactly what the org variable holds — and the agent suites additionally need >= 3.32.0-rc.8 for the /agent/* control plane and TaskDef.runtimeMetadata persistence they exercise. Pinned independently, deliberately. Converging the two is possible once the org variable holds a concrete version (3.32.3 and 3.33.0-rc1 are published to both registries), but that is a bump of a stale rc18 and a rename, not this change. This does move the OSS harness off byte-identical with the other six SDKs, which the previous commit had just restored. Deliberate and temporary: the same edit applies verbatim to java-sdk and python-sdk, held back for now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…image The four "DURABLE + *" cases in WorkflowExecutor.test.ts have been failing on sdkdev across Node 20/22/24 (shard 1/3) with REGION_DURABLE consistency requested but region replication is not enabled/configured on this node Nothing changed in this repo: the cases were named "DURABLE + *" but had always passed Consistency.REGION_DURABLE. They went red when sdkdev picked up a server build carrying the active/active work (orkes-conductor ea0a37ad74, "region-durable workflow start"). Before it, REGION_DURABLE was an inert enum value that fell through the same scheduleWorkflow branch as DURABLE, so the flag was accepted and silently ignored; after it, a node with no replicator wired rejects the start outright rather than appearing to honour a guarantee it cannot provide. The server is right — sdkdev is a single-node instance not meant to have replication — and the tests were green under false pretences: they asked for cross-region durability, got a plain local start, and asserted success. So: - Rename the four cases to "REGION_DURABLE + *", matching what they request, and gate them on CONDUCTOR_REGION_DURABLE_ENABLED via a new testForRegionDurable helper. Explicit opt-in rather than catching the 500 and skipping: auto-skip would restore exactly the silent green that hid this for months, and would mask a genuine replication regression on a cluster where it is meant to work. Same gate name as the java-sdk fix (orkes-io/java-sdk#167) so both SDKs configure identically. - Add four real "DURABLE + *" cases. Consistency.DURABLE appeared nowhere in the suite despite being the default level, and it takes a different executor path than SYNCHRONOUS (async decider queue via scheduleWorkflow, not an inline decide), so the return strategies are worth covering against it. This is not new coverage so much as restored: it is what the REGION_DURABLE cases were really exercising before the server started honouring the flag. Also pins the OSS Conductor image instead of tracking `latest`, so an upstream release cannot turn the OSS job red on an unrelated PR — the same class of surprise as the sdkdev upgrade above. The tag comes from the E2E_TEST_OSS_CONDUCTOR_VERSION org variable or a workflow_dispatch input, with a preflight step that fails with an actionable message when neither resolves, and scripts/run-integration-oss.sh grows a --version flag (defaulting to `latest`) for local runs. Net effect on CI: sdkdev goes from 4 failures to 4 skips plus 4 new passing cases. No CI configuration needs the new variable — gated-off is correct for a single-region server, and java-sdk likewise sets it in no workflow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three gaps against the OSS harness the other six SDKs share (ruby, python, go, csharp, java, rust), which this repo's variant was derived from without carrying across. Each change below is byte-identical to those, so the seven stay diffable. Fork PRs. GitHub withholds org/repo variables from pull_request runs on forks exactly as it withholds secrets, so vars.E2E_TEST_OSS_CONDUCTOR_VERSION is always "" for an outside contributor (observed in csharp-sdk#178). Resolving in the job `env` and null-checking therefore failed every fork PR with a config error the contributor has no power to fix — on a job that needs no secrets at all, only a tag, since it runs a local docker-compose stack against localhost. Resolve in a step writing $GITHUB_ENV instead, so the two ways the tag can come back empty get different treatment: a fork PR pins FORK_PR_FALLBACK_VERSION and keeps running, anything else still fails loudly rather than silently drifting onto the pin. The pin is 3.32.3, matching all six. Server logs on failure. cleanup() ran `compose down -v` without dumping anything, and the only `compose logs` call sat inside the health-wait loop — so logs were captured when the server never came up, but lost whenever the server started fine and the tests then failed. `-v` drops the volumes too, so there was no recovering them afterwards. Local runs were thus strictly worse than CI, which already dumps via an `if: failure()` step. Dump from the EXIT trap on any nonzero status, before the teardown and before the --keep-up early return. `local status=$?` is first so it reflects the failing command; `set -euo pipefail` (already present) carries a failing jest run through the tee pipeline so the trap sees it. The health-loop dump is removed as now redundant — its `exit 1` reaches the same trap — which also brings this script to zero in-loop dumps like the others. Image freshness. `docker compose up` only pulls when the image is missing locally, so a cached mutable tag — `latest`, or a re-pushed rc — was silently reused and nothing reported which image actually ran. Pinning the tag does not help if a stale layer is already cached under that name. Pull conductor-server unconditionally and echo the resolved tag first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review feedback on #176. Three separate points, all about the Conductor OSS image tag having more than one home. One tag, one home. The tag was written twice: FORK_PR_FALLBACK_VERSION in the integration-tests-oss job, and `latest` as the script's default. A local run therefore could not reproduce a CI failure, which is the whole point of the script existing. Rather than teach one of them to read the other, make the `image:` line of docker-compose-oss.yaml the only place it is written and let both fall through to it: the script no longer applies a default at all and exports OSS_CONDUCTOR_VERSION only when --version actually supplied one, and the fork-PR branch of the resolve step now leaves the variable unset instead of pinning its own copy. Fork CI and a plain local run land on the identical image by taking the identical path, with no YAML parsing on either side and one hardcode removed rather than a mechanism added. The non-fork empty case still fails loudly. The script's "Using ..." and "Pulling ..." lines now come from `compose config --images conductor-server` instead of reconstructing the tag, so they stay honest whether it came from the flag or the compose default. Worth stating plainly: E2E_TEST_OSS_CONDUCTOR_VERSION is currently set to `latest` at the org level, so the pin this series added is nominal until someone sets it to a real version. The compose default is what actually holds the line today, on fork PRs and locally. Pull in CI. The script pulls before `up`, CI did not. On a GitHub-hosted runner the VM is ephemeral and starts with no cached copy of this image, so `up` pulls anyway and the step is redundant today — kept regardless, because it costs no extra network pull (`up` then finds the image locally), it splits "couldn't pull the image" from "the stack didn't come up" into two distinct red steps, and it is what stops a mutable tag going stale the day this job moves to a self-hosted runner with a warm Docker daemon. It also prints the tag in use, which matters now that a fork PR's tag is not spelled out in the workflow. agent-e2e's near-namesake. CONDUCTOR_OSS_VERSION there reads like it should be the same knob and is not, so say why at the definition rather than leave the next reader to work it out. That one is a Maven Central artifact version for the conductor-server boot JAR; this one is a Docker Hub image tag. Different registries publish different version sets — Maven has no `latest`, which is exactly what the org variable holds — and the agent suites additionally need >= 3.32.0-rc.8 for the /agent/* control plane and TaskDef.runtimeMetadata persistence they exercise. Pinned independently, deliberately. Converging the two is possible once the org variable holds a concrete version (3.32.3 and 3.33.0-rc1 are published to both registries), but that is a bump of a stale rc18 and a rename, not this change. This does move the OSS harness off byte-identical with the other six SDKs, which the previous commit had just restored. Deliberate and temporary: the same edit applies verbatim to java-sdk and python-sdk, held back for now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chrishagglund-ship-it
force-pushed
the
fix/skip-region-durable-tests-and-improve-oss-version-pin
branch
from
September 10, 2026 17:31
02fd109 to
5227051
Compare
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's failing
Four tests in
WorkflowExecutor.test.tshave been red onintegration v5 sdkdev(shard 1/3, all of Node 20/22/24):
Note the mismatch: the test says
DURABLE, the server saysREGION_DURABLE.Why
The cases were named
DURABLE + *but had always passedConsistency.REGION_DURABLE— the section comment even said// REGION_DURABLE consistency tests. Nothing in this repo changed. They went redwhen sdkdev picked up a server build carrying the active/active work
(
ea0a37ad74, "region-durableworkflow start (synchronous cross-region push, fail-closed)").
Before that change,
REGION_DURABLEwas an inert enum value that fell through thesame
scheduleWorkflowbranch asDURABLE, so the flag was accepted and silentlyignored. After it, a node with no replicator wired rejects the start outright rather
than appearing to honour a guarantee it cannot provide.
The server is right. sdkdev is a single-node instance that is not meant to have
replication. The tests were green under false pretences: they asked for cross-region
durability, got a plain local start, and asserted success.
java-sdk hit this identically and fixed it in
java-sdk#167, which dates
the sdkdev upgrade precisely — 2026-08-11, orkes-conductor 5.3.0 → 5.5.0.
Changes
1. Gate the REGION_DURABLE cases — renamed to
REGION_DURABLE + *to match whatthey actually request, gated behind
CONDUCTOR_REGION_DURABLE_ENABLEDvia a newtestForRegionDurablehelper.Explicit opt-in rather than catch-the-500-and-skip, following java-sdk's reasoning:
auto-skip would restore exactly the silent green that hid this for months, and would
mask a genuine replication regression on a cluster where it is meant to work. Same
env var name as java-sdk so both SDKs configure identically.
2. Add four real
DURABLE + *cases.Consistency.DURABLEappeared nowherein the suite despite being the default level, and it takes a different executor path
than
SYNCHRONOUS:SYNCHRONOUSdecideWithLock+pushToDeciderQueueWithTimeout, inlineDURABLEscheduleWorkflow— async decider queueLess new coverage than restored coverage — it is what the REGION_DURABLE cases were
really exercising all along, back when the server ignored the flag.
3. Write the OSS Conductor image tag in exactly one place. Previously the OSS job
tracked
latest, so an upstream release could turn it red on an unrelated PR — thesame class of surprise as the sdkdev upgrade above. The tag now resolves like this:
E2E_TEST_OSS_CONDUCTOR_VERSIONorg variable, or theworkflow_dispatchinputimage:default inscripts/docker-compose-oss.yaml— GitHub withholdsvars.*from forks exactly as it withholds secretsrun-integration-oss.sh --version <tag>overrides itThe default itself is written once, on the
image:line of the compose file(
3.32.3).run-integration-oss.shapplies no default of its own and exportsOSS_CONDUCTOR_VERSIONonly when--versionactually supplied one, so a plain localrun and a fork CI run reach the identical image by the identical path — which is what
lets the script reproduce a CI failure. This removed a hardcoded fallback rather than
adding a mechanism.
4. Pull the image in CI before starting the stack. Redundant on a GitHub-hosted
runner, whose VM is ephemeral and starts with no cached copy, so
uppulls anyway.Kept regardless: it costs no extra network pull (
upthen finds the image locally),it splits "couldn't pull the image" from "the stack didn't come up" into two distinct
red steps, and it is what stops a mutable tag going stale if this job ever moves to a
self-hosted runner with a warm Docker daemon — the same reason the local script pulls.
It also logs the tag actually in use, which now matters because a fork PR's tag is not
spelled out in the workflow.
Test matrix after this change
SYNC + *×4SYNCHRONOUSDURABLE + *×4DURABLEREGION_DURABLE + *×4REGION_DURABLEsdkdev goes from 4 failures → 4 skips + 4 new passing cases.
Review feedback
All three comments are addressed in
5227051:latest, so the script won't reproduce a CI failurePull Conductor OSS imagestepagent-e2e.ymlalready calls thisCONDUCTOR_OSS_VERSION— use one name, or say why they're separateagent-e2e.yml. They are separate on purpose: that one is a Maven Central artifact version for theconductor-serverboot JAR, this one a Docker Hub image tag. The two registries publish different version sets — Maven has nolatest— and the agent suites carry a floor this job does not (≥3.32.0-rc.8, for the/agent/*control plane andTaskDef.runtimeMetadatapersistence). Converging them is possible once the org variable holds a concrete version, but that is a rename plus a bump of a stale pin, not this PR.Notes for reviewers
CONDUCTOR_REGION_DURABLE_ENABLED. Gated-off is thecorrect state for a single-region server; java-sdk likewise sets it in no workflow.
Set it only against a target with cross-region replication configured.
E2E_TEST_OSS_CONDUCTOR_VERSIONis currently set tolatestat the org level.So the pin in change 3 is nominal for normal CI runs until someone sets that variable
to a real version — today it is the compose default that actually holds the line, on
fork PRs and locally. The variable must also have this repo in its repository access
policy, or a non-fork run fails the preflight step by design.
waitForSecondstimeout the execute endpointreturns
toWorkflowRun(...), which never setsresponseType— so a timeout surfacesas a confusing
undefinedmismatch rather than "it timed out." Affects all 12 combosequally and predates this PR; worth a separate fix.