diff --git a/docs/project/2026-09-18-upstream-v1.2.0-integration-plan.md b/docs/project/2026-09-18-upstream-v1.2.0-integration-plan.md index febb6d80..acf09458 100644 --- a/docs/project/2026-09-18-upstream-v1.2.0-integration-plan.md +++ b/docs/project/2026-09-18-upstream-v1.2.0-integration-plan.md @@ -206,29 +206,55 @@ Part (a) rollout: file-server /health 200; tool-call-server healthy with redis true; egress-gateway /health 200. -Part (a) e2e in stg: blocked, no credential-free path. - -- Every authenticated route returns 401 without a bearer token; an unknown - path also returns 401, so auth precedes routing. -- The deployed revision verifies EdDSA JWTs from issuer librechat with - audience codeapi, single-tenant legacy. Its JWKS advertises one key, - kid codeapi-stg-2026-06-29. -- No synthetic token is configured in stg (CODEAPI_SYNTHETIC_ACCESS_TOKEN is - absent from both api deployments), and no trust-entry override is present - (CODEAPI_JWT_TRUST_ENTRIES_JSON is unset). -- The private signing key is not reachable: the cluster secret carries only - the public JWKS, Infisical has no readable codeapi profile, and Key Vault - secret listing is forbidden for the available identities. -- Stg also has no sandbox capacity to complete an execute: the sandbox-runner - and pool-waker deployments are scaled to zero, the sandboxspot pool is - empty, and there is no KVM node. -- Consequence: an authenticated POST /v1/exec cannot be performed in stg - without an stg signing key or an enabled synthetic token. The - credential-free proof (health surface plus the auth-gate matrix) is what - this environment can demonstrate. -- Note: the deployed revision on main does support CODEAPI_JWT_TRUST_ENTRIES - _JSON, so enabling a synthetic token or a trust entry is a configuration - change rather than a rebuild. +Part (a) e2e in stg: core execute PASSES; artifact delivery is a pre-existing +stg storage defect. + +Correction: an earlier note in this section recorded the e2e as blocked with no +reachable signing key and no sandbox capacity. Both were wrong. The signing key +is reachable through Infisical, and the sandbox pool wakes on demand. The +corrected live evidence follows. + +Credential path (no secret values recorded anywhere): + +- Infisical project codeapi (stg, https://inf.stg.df-app.ch) holds + CODEAPI_JWT_PRIVATE_JWK_JSON, an OKP/Ed25519 key with kid + codeapi-stg-2026-06-29. It derives exactly the deployed JWKS verifier key + held in the codeapi-secrets secret. No infra change was needed. +- Tokens are minted from that key with header {alg: EdDSA, typ: JWT, kid: + codeapi-stg-2026-06-29} and claims sub, jti, iat, nbf, exp, iss librechat, + aud codeapi, principal_source librechat_jwt, auth_context_hash. The deployed + verifier accepts max TTL 300s, issuer librechat, audience codeapi, alg EdDSA, + tenant legacy. + +Core execute result (POST /v1/exec through the deployed codeapi-api): + +- print(123) returns HTTP 200, stdout "123", code 0, wall_time about 48ms. +- Cold start works: the KEDA wake path scaled codeapi-sandbox-pool-waker and + codeapi-sandbox-runner from zero and a sandboxspot KVM node came up. The + runner endpoint became ready after roughly three to four minutes. Early 502 + connect ECONNREFUSED responses were only the cold runner, not a defect. + +Artifact delivery result and root cause: + +- An artifact write (open of stg-artifact.txt for writing) returns HTTP 200 with + stdout "cwd= /mnt/data" then "wrote True" and code 0, but artifact_delivery + reports code artifact_delivery_failed, attempted 1, delivered 0, failed 1, and + files is empty. +- Root cause is the stg SeaweedFS instance, not this repository revision. The + single all-in-one seaweedfs pod registers one volume server at Max=8 volumes. + Volumes 1-7 serve the default collection and volume 8 serves trypost-media, so + the codeapi-files collection has zero writable volumes. The master logs "No + writable volumes and no free volumes left for collection codeapi-files" and + "Not enough data nodes found", and each S3 PutObject returns 500, which the + file-server turns into ten retries and a failed delivery. +- PRD is unaffected: its codeapi-files collection already owns volume 8 and its + file-server shows zero retryable-500 lines. The stg instance simply never + allocated a writable volume to codeapi-files. +- This is an environment and storage provisioning gap predating the rollout. It + is outside the scope of the v1.2.0 integration, which verifies that the + deployed revision behaves correctly. Allocating a SeaweedFS writable volume + for codeapi-files, or raising the volume max, is the separate remediation. + Part (b) dependabot dispositions and PR #21: