run integration(e2e) tests against oss server for PRs

[chrishagglund-ship-it]

Runs the integration (e2e) spec suite against both Conductor server families in CI: a Conductor OSS stack spun up in-job, and the Orkes cloud dev server.

Why

The suite only ever ran against Orkes cloud, on pushes to main, with continue-on-error: true. That combination meant it protected nothing: failures were green, and two examples had in fact never passed at all (see Bugs this surfaced). This PR gets the suite running against OSS on every PR, and fixes the cloud half so that it actually gates.

What changed

1. New integration-tests-oss CI job

Spins up Conductor OSS + Postgres via scripts/docker-compose-oss.yaml, waits for /health, and runs the full spec suite against it unauthenticated. Needs no secrets, so it runs on every push and every PR, and is not continue-on-error.

The image tag comes from the E2E_TEST_OSS_CONDUCTOR_VERSION org variable, with a workflow_dispatch input to override it. The job hard-fails if neither resolves — that's deliberate: if someone unsets the org variable, all SDK CI should break loudly rather than silently test nothing. (Fork PRs are the one exception, since they cannot read the variable at all — see section 7.)

scripts/run-integration-oss.sh runs the identical stack locally (--keep-up, --version <tag>, -- <rspec args>). The compose file and script mirror the equivalents in python-sdk, go-sdk, java-sdk, javascript-sdk, csharp-sdk and rust-sdk.

2. Fixed the existing cloud integration-test job

This job was in scope for the ticket and was already supposed to be working. It wasn't:

• Dropped continue-on-error: true. The suite now has to pass. This is the point of having it.
• Widened the trigger from push + refs/heads/main to push, so develop is covered too.
• Moved CONDUCTOR_SERVER_URL and CONDUCTOR_AUTH_KEY from secrets to vars. Neither is a secret — one is a hostname, the other is a key identifier. Treating them as secrets bought nothing and made them unreadable and awkward to manage. CONDUCTOR_AUTH_SECRET remains a secret, because it is one.

3. Bugfix: scheduler pause/resume now works on both server families

SchedulerResourceApi#pause_schedule / #resume_schedule sent GET unconditionally, which fails outright against OSS. The two families map these two per-schedule routes differently:

	pause / resume
OSS Conductor	PUT only (@PutMapping)
Orkes Conductor	GET and PUT since the dual @RequestMapping(method = {GET, PUT}) added 2026-07; GET only before that

There is no single verb that works everywhere, so the SDKs settled on PUT first, falling back to GET on a 405 — and only on a 405; any other status propagates untouched. This matches python-sdk, go-sdk, javascript-sdk, csharp-sdk and rust-sdk.

This is a bugfix, not a breaking change. The shipped GET-only behavior is broken against OSS, and every Orkes server that accepted GET also accepts PUT. pause_all_schedules / resume_all_schedules are untouched — both families map those admin endpoints GET.

New spec/conductor/http/api/scheduler_resource_api_spec.rb (10 examples) pins the whole contract: PUT-first, 405→GET on the same path, non-405 propagates without a fallback, no dialect memoization (every call re-attempts PUT), and the admin endpoints staying GET.

4. Bugs this surfaced

• Conductor::Configuration::AuthenticationSettings raises NameError. The class is defined directly under Conductor; that nested path has never resolved. integration_helper.rb used it, which broke the three spec files that route through IntegrationHelper.configuration — metadata_spec, workflow_spec, worker_e2e_spec — invisible under continue-on-error. The other seven build their own Conductor::Configuration, whose constructor already resolves auth from the same two env vars, so they were unaffected. Also fixed in RactorTaskRunner's in-Ractor config rebuild (a live failure) and in the Conductor / OrkesClients doc comments, which were telling users to write the broken form.
• Two examples referenced the pre-0.1.0 workflow builder. Conductor::Workflow::ConductorWorkflow, SimpleTask and SetVariableTask were removed in the DSL redesign, so orkes_spec.rb and worker_e2e_spec.rb were raising NameError. Ported to the Conductor.workflow DSL.

5. OSS capability gating

Specs covering APIs that OSS genuinely does not implement now skip themselves via IntegrationHelper.oss? (driven by CONDUCTOR_SERVER_TYPE, documented in that file). Each skip message records the empirically-confirmed gap: Authorization/RBAC, Schema registry, Integration Hub, Prompt templates, schedule tags, secret writes, update_workflow_state, and queue config. Nothing is skipped on the cloud side.

Where behavior only differs rather than being absent, the specs assert the difference instead of skipping — e.g. secrets: reads are verified against an env-seeded secret and writes are asserted to return 501 from OSS's read-only SecretsDAO.

6. Portable search query syntax

Workflow and task search used Lucene field:value syntax, which OSS's default Postgres-backed indexing cannot parse. Switched to the portable field = "value" form, which both families accept.

These two examples previously only asserted expect(results).not_to be_nil — a query the server parses but cannot match returns 200 with zero rows, so a syntax regression was invisible. They now poll until rows appear and assert the rows actually match the expected workflowType/taskType, and the task-search group creates and polls its own task so it doesn't depend on spec ordering.

7. Follow-ups from pre-review

• RestClient now retries Faraday::ConnectionFailed. faraday-retry's DEFAULT_EXCEPTIONS omits it, and it is what the net_http_persistent adapter raises for Errno::ECONNRESET — a write to a pooled socket the peer closed first. net-http-persistent retries stale sockets itself, but only for idempotent requests, so every GET was silently protected while POSTs surfaced as a hard ApiError(status: 0). That is what failed this branch's own cloud run (34265187276 attempt 1, in register_task_def). This does mean a POST is now retried on a connection failure. The existing config already retried POST on 500/502/503/504, where the server definitely received the request; a reset-on-write means it never did, so this is the safer of the two. spec/conductor/http/rest_client_spec.rb pins it, including an example asserting the request is attempted 4 times rather than 1.
• Workflow-level concurrency, superseding an in-flight run when a new commit lands on the same ref, matching python-sdk and go-sdk. More than a runner-time saving here: the cloud job mutates shared state on the dev tenant (scheduler_spec calls pause_all_schedules / resume_all_schedules, which are not scoped to a test_id), so two runs of one branch overlapping will fight each other. The group is ref-scoped, so concurrent main and develop pushes are still not serialized.
• Fork PRs no longer hard-fail the OSS job. GitHub withholds org variables from pull_request runs on forks exactly as it withholds secrets, so E2E_TEST_OSS_CONDUCTOR_VERSION resolves empty for an outside contributor. A fork PR now pins a version and keeps running; a missing variable on a first-party run still fails loudly, per the note in section 1. (csharp-sdk#178 drops that guard entirely in favor of a pinned literal — the split is deliberate here.)
• The Task Search group now unregisters what it registers. Its before hook creates a TaskDef and a WorkflowDef per example and previously only terminated the execution, orphaning both definitions on the shared tenant on every run.

Test results

Job	Result
Integration Tests (OSS)	137 examples, 0 failures, 34 pending
Integration Tests (cloud)	137 examples, 0 failures, 6 pending
Unit	357 examples, 0 failures
RuboCop	154 files, no offenses

Before merge

• Remove e2e-against-conductor-with-local-script from on.push.branches in .github/workflows/ci.yml (kept only so this branch's pushes exercise the cloud job pre-merge; marked with a TODO).