fix(deploy): use development_oidc env for OpenShift JWT auth

[jsell-rh]

Problem

The deploy/openshift overlay enables JWT (--enable-jwt=true) but ran the api-server in the default development environment. That environment's OverrideConfig() force-disables Auth.EnableJWT after flag parsing, so the flag is silently clobbered. REST requests then reach RBAC with no identity and return 401 — the console login loop seen on every OpenShift+Keycloak instance built from this overlay.

Fix

Set API_ENV=development_oidc — the purpose-built environment (e_development_oidc.go) for a self-contained Keycloak:

• leaves EnableJWT to the flags (no hard override), so --enable-jwt=true takes effect
• keeps the OCM/Red Hat AMS client mocked (enable-mock=true default), so the api-server does not require sso.redhat.com secret files and does not crash-loop

This replaces the need for a production + --enable-mock=true workaround, which fought production's enable-mock=false instead of using the environment built for this case.

Also re-declares HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR: the JSON6902 add on the env array replaces the whole array, so without this the api-server loses the control-plane provisioner address.

Alignment

Mirrors the deploy/kind overlay and the hypershell-gitops jwt-enforce component, both of which already use development_oidc.

Test

• kustomize build --load-restrictor LoadRestrictionsNone deploy/openshift renders API_ENV: development_oidc, no --enable-mock flag
• Verified live on hysh-ibm-01 (ROKS): JWT + gateway RBAC enforced, console login succeeds, no AMS secret dependency

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (2)

• do-not-merge/work-in-progress
• do-not-merge/hold

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: bdf2309a-10f5-4471-8ba7-f41e69199de7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[jsell-rh]

Amber review

Status: Complete

Verdict

COMMENT — The root-cause analysis is correct and the fix is well-targeted: development's OverrideConfig() hard-sets Auth.EnableJWT = false after flag parsing, so --enable-jwt=true is silently clobbered, while development_oidc leaves EnableJWT to the flags and keeps the AMS client mocked. One consistency gap in the same env-array replacement (DB_SSLMODE) is worth confirming, and the OpenShift auth patch overlaps materially with another in-flight PR.

I verified the claims against the code:

• e_development.go OverrideConfig() sets c.Auth.EnableJWT = false (hard override after flags) — confirmed.
• e_development_oidc.go OverrideConfig() does not touch EnableJWT, and its Flags() default enable-jwt=true, enable-mock=true — confirmed, so --enable-jwt=true sticks and AMS stays mocked without sso.redhat.com secrets.
• The base main container env (deploy/base/api-server.yaml) is HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR and DB_SSLMODE; the JSON6902 op: add on .../containers/0/env replaces the whole array — confirmed (this replacement already existed pre-PR).

Findings

[Major] The env-array replacement still drops DB_SSLMODE. The base main container sets DB_SSLMODE: require and its command references --db-sslmode=$(DB_SSLMODE) (deploy/base/api-server.yaml). This PR correctly recognizes that the JSON6902 add replaces the entire env array and re-declares HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR, but DB_SSLMODE is not re-added. If the overlay renders as I read it, $(DB_SSLMODE) in the serve container will be left unresolved (K8s leaves unknown $(VAR) references as the literal string), producing --db-sslmode=$(DB_SSLMODE). Your live verification on hysh-ibm-01 (console login succeeding, which needs the serve container's DB connection) suggests it may resolve some other way — please confirm the rendered/resolved sslmode value. If it does not resolve, re-add DB_SSLMODE to the replacing env array for the same reason you re-added the provisioner address. Confidence: Medium.

[Minor] development_oidc is a dev-named environment used for a real OpenShift cluster. This is a pragmatic choice (it is the only environment that leaves EnableJWT to flags while mocking AMS), and it mirrors deploy/kind, but the name invites confusion for a production-facing overlay, and development_oidc.Flags() also defaults api-server-hostname=localhost. Worth a one-line note in the overlay (already partly covered by the added comment) or a follow-up to introduce a purpose-named environment. Confidence: Medium.

Cross-PR coordination

Another open pull request rewrites the same OpenShift api-server auth patch in deploy/openshift/kustomization.yaml to enforce JWT issuer/audience: it adds a JWT_ISSUER env entry plus --jwt-issuer and --jwt-audience=hypershell-frontend flags, and it upgrades the api-server to an rh-trex-ai version whose JWT verification fails closed. That work and this PR both edit the single JSON6902 op: add that replaces the entire container env array and both append to the same command list, so they cannot be merged mechanically: whoever merges second must fold every env entry (API_ENV, HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR, JWK_CERT_URL, JWT_ISSUER, RBAC_ENFORCE, RBAC_SERVICE_ACCOUNTS) into the one replacing array, or a silently dropped entry will disappear. There is also an ordering dependency: this PR enables JWT via development_oidc + --enable-jwt=true, while the other PR makes verification require issuer/audience and fail closed; if this overlay turns JWT on without the issuer/audience flags once that rh-trex-ai upgrade lands, tokens can be rejected. Maintainers should decide the merge order and the combined final auth configuration for this overlay.

Convention checklist

Convention	Result
No panic() in production code	N/A (config only)
Config separate from code	Pass
Image references consistent across manifests	N/A
Conventional commit message	Pass
Env-array replacement preserves base values	Fail (DB_SSLMODE)

Findings Summary (highest severity first):

1. [Major] Env-array replacement re-adds the provisioner address but drops DB_SSLMODE, which the base serve command references via --db-sslmode=$(DB_SSLMODE) — Config Consistency (L87-94)
2. [Minor] Dev-named development_oidc environment used for a real cluster; api-server-hostname defaults to localhost — Design/Clarity (L87-88)

[jsell-rh]

Verdict

COMMENT — The root-cause analysis is correct and the fix is well-targeted: development's OverrideConfig() hard-sets Auth.EnableJWT = false after flag parsing, so --enable-jwt=true is silently clobbered, while development_oidc leaves EnableJWT to the flags and keeps the AMS client mocked. One consistency gap in the same env-array replacement (DB_SSLMODE) is worth confirming, and the OpenShift auth patch overlaps materially with another in-flight PR.

I verified the claims against the code:

• e_development.go OverrideConfig() sets c.Auth.EnableJWT = false (hard override after flags) — confirmed.
• e_development_oidc.go OverrideConfig() does not touch EnableJWT, and its Flags() default enable-jwt=true, enable-mock=true — confirmed, so --enable-jwt=true sticks and AMS stays mocked without sso.redhat.com secrets.
• The base main container env (deploy/base/api-server.yaml) is HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR and DB_SSLMODE; the JSON6902 op: add on .../containers/0/env replaces the whole array — confirmed (this replacement already existed pre-PR).

Findings

[Major] The env-array replacement still drops DB_SSLMODE. The base main container sets DB_SSLMODE: require and its command references --db-sslmode=$(DB_SSLMODE) (deploy/base/api-server.yaml). This PR correctly recognizes that the JSON6902 add replaces the entire env array and re-declares HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR, but DB_SSLMODE is not re-added. If the overlay renders as I read it, $(DB_SSLMODE) in the serve container will be left unresolved (K8s leaves unknown $(VAR) references as the literal string), producing --db-sslmode=$(DB_SSLMODE). Your live verification on hysh-ibm-01 (console login succeeding, which needs the serve container's DB connection) suggests it may resolve some other way — please confirm the rendered/resolved sslmode value. If it does not resolve, re-add DB_SSLMODE to the replacing env array for the same reason you re-added the provisioner address. Confidence: Medium.

[Minor] development_oidc is a dev-named environment used for a real OpenShift cluster. This is a pragmatic choice (it is the only environment that leaves EnableJWT to flags while mocking AMS), and it mirrors deploy/kind, but the name invites confusion for a production-facing overlay, and development_oidc.Flags() also defaults api-server-hostname=localhost. Worth a one-line note in the overlay (already partly covered by the added comment) or a follow-up to introduce a purpose-named environment. Confidence: Medium.

Cross-PR coordination

Another open pull request rewrites the same OpenShift api-server auth patch in deploy/openshift/kustomization.yaml to enforce JWT issuer/audience: it adds a JWT_ISSUER env entry plus --jwt-issuer and --jwt-audience=hypershell-frontend flags, and it upgrades the api-server to an rh-trex-ai version whose JWT verification fails closed. That work and this PR both edit the single JSON6902 op: add that replaces the entire container env array and both append to the same command list, so they cannot be merged mechanically: whoever merges second must fold every env entry (API_ENV, HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR, JWK_CERT_URL, JWT_ISSUER, RBAC_ENFORCE, RBAC_SERVICE_ACCOUNTS) into the one replacing array, or a silently dropped entry will disappear. There is also an ordering dependency: this PR enables JWT via development_oidc + --enable-jwt=true, while the other PR makes verification require issuer/audience and fail closed; if this overlay turns JWT on without the issuer/audience flags once that rh-trex-ai upgrade lands, tokens can be rejected. Maintainers should decide the merge order and the combined final auth configuration for this overlay.

Convention checklist

Convention	Result
No panic() in production code	N/A (config only)
Config separate from code	Pass
Image references consistent across manifests	N/A
Conventional commit message	Pass
Env-array replacement preserves base values	Fail (DB_SSLMODE)

Findings Summary (highest severity first):

1. [Major] Env-array replacement re-adds the provisioner address but drops DB_SSLMODE, which the base serve command references via --db-sslmode=$(DB_SSLMODE) — Config Consistency (L87-94)
2. [Minor] Dev-named development_oidc environment used for a real cluster; api-server-hostname defaults to localhost — Design/Clarity (L87-88)

[jsell-rh]

[Major] DB_SSLMODE is still dropped by this env-array replacement. You correctly re-add HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR here because the JSON6902 op: add on .../containers/0/env replaces the whole array, but the base main container also defines DB_SSLMODE: require and its command uses --db-sslmode=$(DB_SSLMODE) (deploy/base/api-server.yaml). With DB_SSLMODE unset, Kubernetes leaves $(DB_SSLMODE) as a literal string, yielding --db-sslmode=$(DB_SSLMODE). Your live test suggests it works, so please confirm the resolved sslmode; otherwise re-add DB_SSLMODE here for the same reason you re-added the provisioner address. (Confidence: Medium.)

[jsell-rh]

[Minor] Using the dev-named development_oidc environment for a real OpenShift cluster is pragmatic (only env that leaves EnableJWT to flags while mocking AMS, mirrors deploy/kind), but note development_oidc.Flags() also defaults api-server-hostname=localhost. Consider a follow-up to introduce a purpose-named environment for cluster deployments. (Confidence: Medium.)

[squizzi]

Closing, this is addressed in #232 with a very similar method